{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "W6_Tutorial2.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "machine_shape": "hm",
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/CIS-522/course-content/blob/main/tutorials/W06_ConvNets/W6_Tutorial2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X2kPrtVyXtaP"
      },
      "source": [
        "# CIS-522 Week 6 Part 2\r\n",
        "# Convolutional Neural Networks \r\n",
        "\r\n",
        "__Instructor__: Konrad Kording\r\n",
        "\r\n",
        "__Content creators:__ Hmrishav Bandyopadhyay, Rahul Shekhar, Tejas Srivastava\r\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p2DuJddRXxYY"
      },
      "source": [
        "---\r\n",
        "# Tutorial Objectives\r\n",
        "At the end of this tutorial, we will be able to:\r\n",
        "- understand how CNN's work end to end\r\n",
        "- write functions to train and test Deep Leaning models in PyTorch\r\n",
        "- identify the various broad uses of Convnets \r\n",
        " "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e6GnDC12UtUD",
        "cellView": "form"
      },
      "source": [
        "#@markdown What is your Pennkey and pod? (text, not numbers, e.g. bfranklin)\n",
        "\n",
        "my_pennkey = '' #@param {type:\"string\"}\n",
        "my_pod = 'Select' #@param ['Select', 'euclidean-wombat', 'sublime-newt', 'buoyant-unicorn', 'lackadaisical-manatee','indelible-stingray','superfluous-lyrebird','discreet-reindeer','quizzical-goldfish','astute-jellyfish','ubiquitous-cheetah','nonchalant-crocodile','fashionable-lemur','spiffy-eagle','electric-emu','quotidian-lion']"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vgOQxVS1X2dB"
      },
      "source": [
        "---\r\n",
        "# Setup\r\n",
        "\r\n",
        "\r\n",
        "[Here](https://drive.google.com/file/d/1okbxJdaKwi1klnkSkrYoBriDmng_wk0q/view?usp=sharing) are the slides for today's videos (in case you want to take notes). **Do not read them now.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fZue0ldzOv_J",
        "cellView": "form"
      },
      "source": [
        "#@title Installations\n",
        "# Install Kipoi model zoo dependencies for use in genomics section\n",
        "# Kipoi is a library that will be used to install and run the DeepSEA model later in the notebook\n",
        "%%capture\n",
        "!pip install -q pyYaml>=5.1\n",
        "!pip install -q kipoi\n",
        "!pip install -q kipoiseq pybedtools"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c7JFbrL9doH2"
      },
      "source": [
        "# imports\r\n",
        "import random\r\n",
        "import pathlib\r\n",
        "\r\n",
        "import torch\r\n",
        "import numpy as np\r\n",
        "import matplotlib.pyplot as plt\r\n",
        "import kipoi\r\n",
        "\r\n",
        "import torch.nn as nn\r\n",
        "import torch.optim as optim\r\n",
        "import torch.nn.functional as F\r\n",
        "import torchvision.transforms as transforms\r\n",
        "from torchvision.datasets import ImageFolder\r\n",
        "import torchvision.datasets as datasets\r\n",
        "from torch.utils.data import DataLoader, TensorDataset\r\n",
        "from torchvision.utils import make_grid\r\n",
        "from IPython.display import HTML, display\r\n",
        "\r\n",
        "from tqdm.notebook import tqdm, trange\r\n",
        "from time import sleep"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "XbS_xC8NgPZC"
      },
      "source": [
        "# @title Figure Settings\n",
        "%config InlineBackend.figure_format = 'retina'\n",
        "%matplotlib inline \n",
        "plt.style.use(\"https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle\")\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2ZVpX4SYr_sU"
      },
      "source": [
        "---\r\n",
        "# Section 1: Recap\r\n",
        "\r\n",
        "In the last tutorial we looked at, \r\n",
        "- how convolution operation works\r\n",
        "- significance of Convolutions\r\n",
        "- maxpooling and its use\r\n",
        "- and you built you're first CNN!\r\n",
        "\r\n",
        "<br>\r\n",
        "\r\n",
        "*Estimated Completion Time: 12 mins from start of the session*"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LuV3zAQSBKN3",
        "cellView": "form"
      },
      "source": [
        "#@title Video: Discussing Week 6 T1\r\n",
        "import time\r\n",
        "try: t0;\r\n",
        "except NameError: t0=time.time()\r\n",
        "\r\n",
        "from IPython.display import YouTubeVideo\r\n",
        "\r\n",
        "video = YouTubeVideo(id=\"aVmjxdOthzs\", width=854, height=480, fs=1)\r\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\r\n",
        "\r\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WeHMiZ0bsNWe",
        "cellView": "form"
      },
      "source": [
        "# @markdown ## Exercise 1\r\n",
        "# @markdown What will be the output dimension of a single 2D convolution operation on: input of size 300 x 400, a kernel of size (5, 5), stride = 1,  padding = 2 ?\r\n",
        "output_dim = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qwYWn_KNCkGP",
        "cellView": "form"
      },
      "source": [
        "#@title Video: Parameters and Getting a feel of the Dataset\r\n",
        "\r\n",
        "video = YouTubeVideo(id=\"H4SErtYD3qM\", width=854, height=480, fs=1)\r\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\r\n",
        "\r\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dr8gizeWxAL8"
      },
      "source": [
        "---\n",
        "# Section 2: Training a CNN"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yDTUIpYDxPn6"
      },
      "source": [
        "In this section, we will walk you guys through an example of training a convolution net. In the beginning, we will train a CNN simply using convolution layers and maxpool and then observe what the training and validation curves look like. After that, we will add regularization and data augmentation to see what effects they have on the curves and why it is important to incorporate them while training our network.\r\n",
        "<br>\r\n",
        "*Estimated Completion Time: 85 mins from start of the session*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C1-HQXAiyhAU"
      },
      "source": [
        "## 2.1 Understand the Dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sDs3kh4-xpR2"
      },
      "source": [
        "The dataset we are going to use for this task is called Fashion-MNIST. It consists of a training set of 60,000 examples and a test set of 10,000 examples. We further divide the test set into a validation set and a test set (8000 and 2000 resp). Each example is a 28*28 gray scale image, associated with a label from 10 classes. Following are the labels of the dataset:\n",
        "\n",
        "0 T-shirt/top <br>\n",
        "1 Trouser <br>\n",
        "2 Pullover <br>\n",
        "3 Dress <br>\n",
        "4 Coat <br> \n",
        "5 Sandal <br>\n",
        "6 Shirt <br> \n",
        "7 Sneaker <br> \n",
        "8 Bag <br> \n",
        "9 Ankle boot <br> "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p5BbvtUDR5I8",
        "cellView": "form"
      },
      "source": [
        "# @title Utils\n",
        "device =  torch.device('cuda' if torch.cuda.is_available() else 'cpu')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nov2OCVVE2TB",
        "cellView": "form"
      },
      "source": [
        "# @title Getting Fashion-Mnist Data\n",
        "# need to split into train, validation, test \n",
        "def get_fashion_mnist_dataset():\n",
        "    transform = transforms.Compose([\n",
        "        transforms.ToTensor(),\n",
        "        transforms.Normalize((0.1307,), (0.3081,))\n",
        "        ])\n",
        "\n",
        "    train_data = datasets.FashionMNIST(root='./data', download=True, train=True, transform=transform)\n",
        "    \n",
        "    test_data = datasets.FashionMNIST(root='./data',  download=True, train=False, transform=transform)\n",
        "\n",
        "    validation_data, test_data = torch.utils.data.random_split(test_data, [8000, 2000])\n",
        "\n",
        "    return train_data, validation_data, test_data\n",
        "\n",
        "train_data, validation_data, test_data = get_fashion_mnist_dataset()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G_1y5aMJRQy0",
        "cellView": "form"
      },
      "source": [
        "# @title Getting the DataLoaders\n",
        "def get_data_loaders(train_dataset, validation_dataset, test_dataset, batch_size=64):\n",
        "    train_loader = DataLoader(train_dataset, batch_size=batch_size,\n",
        "                         shuffle=True, num_workers=0)\n",
        "    validation_loader = DataLoader(validation_dataset, batch_size=batch_size,\n",
        "                         shuffle=True, num_workers=0)\n",
        "    test_loader = DataLoader(test_dataset, batch_size=batch_size,\n",
        "                         shuffle=True, num_workers=0)\n",
        "    \n",
        "    return train_loader, validation_loader, test_loader\n",
        "\n",
        "train_loader, validation_loader, test_loader = get_data_loaders(train_data, validation_data, test_data)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yzaVJ_Piyff8"
      },
      "source": [
        "Here's some code to visualize the dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CTDOHNZHoqRY"
      },
      "source": [
        "fig, (ax1, ax2, ax3, ax4) = plt.subplots(1, 4)\r\n",
        "ax1.imshow(train_data[0][0].reshape(28, 28), cmap=plt.get_cmap('gray'))\r\n",
        "ax2.imshow(train_data[1][0].reshape(28, 28), cmap=plt.get_cmap('gray'))\r\n",
        "ax3.imshow(train_data[2][0].reshape(28, 28), cmap=plt.get_cmap('gray'))\r\n",
        "ax4.imshow(train_data[3][0].reshape(28, 28), cmap=plt.get_cmap('gray'))\r\n",
        "fig.set_size_inches(18.5, 10.5)\r\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RwMASuKFB4N2"
      },
      "source": [
        "## 2.2 Training Loop"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3qdqljdhCk_U",
        "cellView": "form"
      },
      "source": [
        "#@title Video: Train Loop\r\n",
        "try: t1;\r\n",
        "except NameError: t1=time.time()\r\n",
        "\r\n",
        "video = YouTubeVideo(id=\"StTv7aGCDf0\", width=854, height=480, fs=1)\r\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\r\n",
        "\r\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q2v_LVtVtsNI"
      },
      "source": [
        "At the end of T1, you coded the structure of a CNN. Now, you are going to implement the training loop for a CNN. \n",
        "- Choose the correct criterion \n",
        "- Code up the training part (calculating gradients, loss, stepping forward)\n",
        "- Keep a track of the running loss i.e for each epoch we want to to know the average loss of the batch size. We have already done the same for accuracy for you. \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KXcG75YiEl2o"
      },
      "source": [
        "## Exercise 2.1 Code the Training Loop"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q8W2XmmeEAKS",
        "cellView": "form"
      },
      "source": [
        "# @title Running model on Test Data Function (run me)\n",
        "\n",
        "# just returns accuracy on test data\n",
        "def test(model, device, data_loader):\n",
        "    model.eval()\n",
        "    correct = 0\n",
        "    total = 0\n",
        "    for data in data_loader:\n",
        "        inputs, labels = data\n",
        "        inputs = inputs.to(device).float()\n",
        "        labels = labels.to(device).long()\n",
        "\n",
        "        outputs = model(inputs)\n",
        "        _, predicted = torch.max(outputs, 1)\n",
        "        total += labels.size(0)\n",
        "        correct += (predicted == labels).sum().item()\n",
        "\n",
        "    acc = 100 * correct / total\n",
        "    return acc"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5cQecTtcfWV1",
        "cellView": "form"
      },
      "source": [
        "# @title Plotting (run me)\n",
        "\n",
        "# code to plot loss and accuracy\n",
        "def plot_loss_accuracy(train_loss, train_acc, validation_loss, validation_acc):\n",
        "    epochs = len(train_loss)\n",
        "    fig, (ax1, ax2) = plt.subplots(1, 2)\n",
        "    ax1.plot(list(range(epochs)), train_loss, label='Training Loss')\n",
        "    ax1.plot(list(range(epochs)), validation_loss, label='Validation Loss')\n",
        "    ax1.set_xlabel('Epochs')\n",
        "    ax1.set_ylabel('Loss')\n",
        "    ax1.set_title('Epoch vs Loss')\n",
        "    ax1.legend()\n",
        "\n",
        "    ax2.plot(list(range(epochs)), train_acc, label='Training Accuracy')\n",
        "    ax2.plot(list(range(epochs)), validation_acc, label='Validation Accuracy')\n",
        "    ax2.set_xlabel('Epochs')\n",
        "    ax2.set_ylabel('Accuracy')\n",
        "    ax2.set_title('Epoch vs Accuracy')\n",
        "    ax2.legend()\n",
        "    fig.set_size_inches(15.5, 5.5)\n",
        "    plt.show()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y4t8I4pLCawz"
      },
      "source": [
        "# This cell contains the code for the CNN we will be using in this section.\n",
        "class FMNIST_Net1(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(FMNIST_Net1, self).__init__()\n",
        "        self.conv1 = nn.Conv2d(1, 32, 3, 1)\n",
        "        self.conv2 = nn.Conv2d(32, 64, 3, 1)\n",
        "        self.fc1 = nn.Linear(9216, 128)\n",
        "        self.fc2 = nn.Linear(128, 10)\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = self.conv1(x)\n",
        "        x = F.relu(x)\n",
        "        x = self.conv2(x)\n",
        "        x = F.relu(x)\n",
        "        x = F.max_pool2d(x, 2)\n",
        "        x = torch.flatten(x, 1)\n",
        "        x = self.fc1(x)\n",
        "        x = F.relu(x)\n",
        "        x = self.fc2(x)\n",
        "       \n",
        "        return x"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k4cfJKOwtOXa"
      },
      "source": [
        "def train(model, device, train_loader, validation_loader, epochs):\n",
        "    \n",
        "    ####################################################################\n",
        "    # Fill in missing code below (...),\n",
        "    # then remove or comment the line below to test your function\n",
        "    raise NotImplementedError(\"Define the loss function\")\n",
        "    ####################################################################\n",
        "    criterion = ...\n",
        "    optimizer = torch.optim.SGD(model.parameters(),\n",
        "                              lr=0.01, momentum=0.9)\n",
        "    \n",
        "    train_loss, validation_loss = [], []\n",
        "    train_acc, validation_acc = [], []\n",
        "    for epoch in range(epochs):\n",
        "        model.train()\n",
        "        # keeps track of the running loss\n",
        "        running_loss = 0.\n",
        "\n",
        "        correct, total = 0, 0 \n",
        "        with tqdm(train_loader, unit='batch') as tepoch:\n",
        "            tepoch.set_description('Training: ')\n",
        "            for data, target in tepoch:\n",
        "                data, target = data.to(device), target.to(device)\n",
        "\n",
        "                # COMPLETE CODE FOR TRAINING LOOOP\n",
        "                # 1. Get the model output\n",
        "                # 2. Zero the gradients out\n",
        "                # 3. Get the Loss\n",
        "                # 4. Calculate the gradients\n",
        "                # 5. Update the weights\n",
        "                \n",
        "                ####################################################################\n",
        "                # Fill in missing code below (...),\n",
        "                # then remove or comment the line below to test your function\n",
        "                raise NotImplementedError(\"Update the set_postfix function\")\n",
        "                ####################################################################\n",
        "\n",
        "                \n",
        "                # set loss to whatever you end up naming your variable when \n",
        "                # calling criterion\n",
        "                # for example, loss = criterion(output, target)\n",
        "                # then set loss = loss.item() in the set_postfix function\n",
        "                tepoch.set_postfix(loss=...)\n",
        "                \n",
        "                # get accuracy \n",
        "                _, predicted = torch.max(output, 1)\n",
        "                total += target.size(0)\n",
        "                correct += (predicted == target).sum().item()\n",
        "        \n",
        "        ####################################################################\n",
        "        # Fill in missing code below (...),\n",
        "        # then remove or comment the line below to test your function\n",
        "        raise NotImplementedError(\"Append the train_loss\")\n",
        "        ####################################################################\n",
        "        train_loss.append(...)  # append the loss for this epoch\n",
        "        train_acc.append(correct/total)\n",
        "                \n",
        "        # evaluate on validation data\n",
        "        \n",
        "        model.eval()\n",
        "        running_loss = 0.\n",
        "        correct, total = 0, 0 \n",
        "        with tqdm(validation_loader, unit='batch') as tepoch:\n",
        "            tepoch.set_description('Validation: ')\n",
        "            for data, target in tepoch:\n",
        "                data, target = data.to(device), target.to(device)\n",
        "                optimizer.zero_grad()\n",
        "                output = model(data)\n",
        "                \n",
        "                loss = criterion(output, target)\n",
        "                tepoch.set_postfix(loss=loss.item())\n",
        "                running_loss += loss.item()\n",
        "\n",
        "                # get accuracy \n",
        "                _, predicted = torch.max(output, 1)\n",
        "                total += target.size(0)\n",
        "                correct += (predicted == target).sum().item()\n",
        "        \n",
        "        validation_loss.append(running_loss/len(validation_loader))\n",
        "        validation_acc.append(correct/total)\n",
        "    \n",
        "    return train_loss, train_acc, validation_loss, validation_acc "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UwCUqba1RtXo"
      },
      "source": [
        "# to_remove solution\n",
        "def train(model, device, train_loader, validation_loader, epochs):\n",
        "\n",
        "    criterion = nn.CrossEntropyLoss()\n",
        "    optimizer = torch.optim.SGD(model.parameters(),\n",
        "                              lr=0.01, momentum=0.9)\n",
        "    \n",
        "    train_loss, validation_loss = [], []\n",
        "    train_acc, validation_acc = [], []\n",
        "    for epoch in range(epochs):\n",
        "        model.train()\n",
        "        running_loss = 0.\n",
        "        correct, total = 0, 0 \n",
        "        with tqdm(train_loader, unit='batch') as tepoch:\n",
        "            tepoch.set_description('Training: ')\n",
        "            for data, target in tepoch:\n",
        "                data, target = data.to(device), target.to(device)\n",
        "\n",
        "                # add micro for coding training loop\n",
        "                optimizer.zero_grad()\n",
        "                output = model(data)\n",
        "                \n",
        "                loss = criterion(output, target)\n",
        "                loss.backward()\n",
        "                optimizer.step()\n",
        "                tepoch.set_postfix(loss=loss.item())\n",
        "                running_loss += loss.item()\n",
        "\n",
        "                # get accuracy \n",
        "                _, predicted = torch.max(output, 1)\n",
        "                total += target.size(0)\n",
        "                correct += (predicted == target).sum().item()\n",
        "        \n",
        "        train_loss.append(running_loss/len(train_loader))\n",
        "        train_acc.append(correct/total)\n",
        "                \n",
        "        # evaluate on validation data\n",
        "        model.eval()\n",
        "        running_loss = 0.\n",
        "        correct, total = 0, 0 \n",
        "        with tqdm(validation_loader, unit='batch') as tepoch:\n",
        "            tepoch.set_description('Validation: ')\n",
        "            for data, target in tepoch:\n",
        "                data, target = data.to(device), target.to(device)\n",
        "                optimizer.zero_grad()\n",
        "                output = model(data)\n",
        "                \n",
        "                loss = criterion(output, target)\n",
        "                tepoch.set_postfix(loss=loss.item())\n",
        "                running_loss += loss.item()\n",
        "\n",
        "                # get accuracy \n",
        "                _, predicted = torch.max(output, 1)\n",
        "                total += target.size(0)\n",
        "                correct += (predicted == target).sum().item()\n",
        "        \n",
        "        validation_loss.append(running_loss/len(validation_loader))\n",
        "        validation_acc.append(correct/total)\n",
        "    \n",
        "    return train_loss, train_acc, validation_loss, validation_acc \n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JVJWogQEEr3x"
      },
      "source": [
        "The next cell contains the code for the CNN we will be using in this section."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ssiflQoeFcbX"
      },
      "source": [
        "net = FMNIST_Net1().to(device)\n",
        "train_loss, train_acc, validation_loss, validation_acc = train(net, device, train_loader, validation_loader, 20)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hbfN8Yu6CygD"
      },
      "source": [
        "plot_loss_accuracy(train_loss, train_acc, validation_loss, validation_acc)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vl9w7jsPD6o2"
      },
      "source": [
        "Run the next cell to get the accuracy on the data!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DamEu_3AGLGM"
      },
      "source": [
        "test(net, device, test_loader)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h7jAGOEdo67G",
        "cellView": "form"
      },
      "source": [
        "#@markdown Q. Do you think this network is overfitting? What can you do to combat this?\r\n",
        "overfitting = '' #@param {type:\"string\"}\r\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8Qp5F364E7C2"
      },
      "source": [
        "## 2.3: Regularization\n",
        "\n",
        " "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_1gJsQsaCkfO",
        "cellView": "form"
      },
      "source": [
        "#@title Video: Dropout\r\n",
        "try: t2;\r\n",
        "except NameError: t2=time.time()\r\n",
        "\r\n",
        "video = YouTubeVideo(id=\"V2XwxhZ1SeU\", width=854, height=480, fs=1)\r\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\r\n",
        "\r\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pCMksxJ1FCek"
      },
      "source": [
        "So you spent week 5 learning a bunch of regularization techniques. We have copied over the above CNN model and we want you to add some regularization methods into this network and to check if that helps reduce the overfitting problem. You can try methods other than dropout as well."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rNBNQ_bf04eB"
      },
      "source": [
        "## Exercise 2.2: Adding Regularization"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bDz9rj_MLl-t"
      },
      "source": [
        "# add various regularization methods, feel free to add any and play around! \n",
        "class FMNIST_Net2(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(FMNIST_Net2, self).__init__()\n",
        "        self.conv1 = nn.Conv2d(1, 32, 3, 1)\n",
        "        self.conv2 = nn.Conv2d(32, 64, 3, 1)\n",
        "        self.fc1 = nn.Linear(9216, 128)\n",
        "        self.fc2 = nn.Linear(128, 10)\n",
        "        ####################################################################\n",
        "        # Fill in missing code below (...),\n",
        "        # then remove or comment the line below to test your function\n",
        "        raise NotImplementedError(\"Add regularization layers\")\n",
        "        ####################################################################\n",
        "        self.dropout1 = ...\n",
        "        self.dropout2 = ...\n",
        "        self.batch_norm1 = ...\n",
        "        self.batch_norm2 = ...\n",
        "\n",
        "    def forward(self, x):\n",
        "        ####################################################################\n",
        "        # Now add the layers in your forward pass in appropriate order\n",
        "        # then remove or comment the line below to test your function\n",
        "        raise NotImplementedError(\"Add regularization in the forward pass\")\n",
        "        ####################################################################\n",
        "        x = self.conv1(x)\n",
        "        x = ...\n",
        "       \n",
        "        return x"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vND29krrFgR0"
      },
      "source": [
        "# to_remove solution\n",
        "class FMNIST_Net2(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(FMNIST_Net2, self).__init__()\n",
        "        self.conv1 = nn.Conv2d(1, 32, 3, 1)\n",
        "        self.batch_norm1 = nn.BatchNorm2d(32)\n",
        "        self.conv2 = nn.Conv2d(32, 64, 3, 1)\n",
        "        self.batch_norm2 = nn.BatchNorm2d(64)\n",
        "        self.dropout1 = nn.Dropout(0.25)\n",
        "        self.dropout2 = nn.Dropout(0.5)\n",
        "        self.fc1 = nn.Linear(9216, 128)\n",
        "        self.fc2 = nn.Linear(128, 10)\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = self.conv1(x)\n",
        "        x = self.batch_norm1(x)\n",
        "        x = F.relu(x)\n",
        "        x = self.conv2(x)\n",
        "        x = self.batch_norm2(x)\n",
        "        x = F.relu(x)\n",
        "        x = F.max_pool2d(x, 2)\n",
        "        x = self.dropout1(x)\n",
        "        x = torch.flatten(x, 1)\n",
        "        x = self.fc1(x)\n",
        "        x = F.relu(x)\n",
        "        x = self.dropout2(x)\n",
        "        x = self.fc2(x)\n",
        "        return x"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GlhYXahxI4k-"
      },
      "source": [
        "net2 = FMNIST_Net2().to(device)\n",
        "train_loss, train_acc, validation_loss, validation_acc = train(net2, device, train_loader, validation_loader, 20)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WA5ejC8JJAYi"
      },
      "source": [
        "plot_loss_accuracy(train_loss, train_acc, validation_loss, validation_acc)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5bKEiBmjI-KX"
      },
      "source": [
        "test(net2, device, test_loader)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7WjLdOtz1zXZ",
        "cellView": "form"
      },
      "source": [
        "#@markdown Is the training accuracy slighly reduced from before adding regularization? What point were you able to reduce it till?\r\n",
        "regularize_1 = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mwhD1_0H193U",
        "cellView": "form"
      },
      "source": [
        "#@markdown Why does the validation accuracy start higher than training accuracy?\r\n",
        "regularize_2 = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9t76Dme9nVCs"
      },
      "source": [
        "## 2.4: Data Augmentation\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T1L7yivJCko1",
        "cellView": "form"
      },
      "source": [
        "#@title Video: Data Augmentation\r\n",
        "try: t3;\r\n",
        "except NameError: t3=time.time()\r\n",
        "\r\n",
        "video = YouTubeVideo(id=\"464k_nu-YPo\", width=854, height=480, fs=1)\r\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\r\n",
        "\r\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m6Co5ZgP2MKs"
      },
      "source": [
        "## Exercise: 2.3\r\n",
        "\r\n",
        "One of the final things you learnt in week 5 was how data augmentation can also be used for regularization. Let us now add that to our model via transforms and see if that helps our model generalize better! In the following cell, add the transforms you want in the list augmentation_transforms. We will then run the same network you created in the above exercise (with regularization) and then plot the loss and accuracies.\r\n",
        "\r\n",
        "Here's the link to the list of transforms available in pytorch: https://pytorch.org/docs/stable/torchvision/transforms.html\r\n",
        "\r\n",
        "Note: In the video, Prof. Kording mentioned that you should modify the train loop. There is no need to modify the train loop, just fill in the function below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UV1HNLBpL6tD"
      },
      "source": [
        "def get_augmentation_transforms():\n",
        "    ####################################################################\n",
        "    # Fill in missing code below (...),\n",
        "    # then remove or comment the line below to test your function\n",
        "    raise NotImplementedError(\"Add Transforms\")\n",
        "    ####################################################################\n",
        "    augmentation_trasnforms = [...]\n",
        "\n",
        "    return augmentation_transforms "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i2D_fT4tJQbn"
      },
      "source": [
        "# to_remove solution\n",
        "def get_augmentation_transforms():\n",
        "   \n",
        "    augmentation_trasnforms = [transforms.RandomRotation(10), transforms.RandomHorizontalFlip()]\n",
        "\n",
        "    return augmentation_trasnforms "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a3arstlaJWox"
      },
      "source": [
        "augmentation_transforms = get_augmentation_transforms()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V7QaH8EvnYLh"
      },
      "source": [
        "preprocessing_transforms = [transforms.ToTensor(),transforms.Normalize((0.1307,), (0.3081,))]\n",
        "train_transform = transforms.Compose(augmentation_transforms + preprocessing_transforms)\n",
        "train_data = datasets.FashionMNIST(root='./data', download=True, train=True, transform=train_transform)\n",
        "train_loader, validation_loader, test_loader = get_data_loaders(train_data, validation_data, test_data)\n",
        "net3 = FMNIST_Net2().to(device)\n",
        "train_loss, train_acc, validation_loss, validation_acc = train(net3, device, train_loader, validation_loader, 20)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_QxPeB6io8w8"
      },
      "source": [
        "plot_loss_accuracy(train_loss, train_acc, validation_loss, validation_acc)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tGR2GP-qpAia"
      },
      "source": [
        "test(net3, device, test_loader)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dDTSFGJL2YB2",
        "cellView": "form"
      },
      "source": [
        "#@markdown Did the training accuracy reduce further? Is the model still overfitting?\r\n",
        "augmentation = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YGIa68kgJsW8"
      },
      "source": [
        "Great! You just trained what may be your very first CNN. You added regularizationa and data augmentation in order to get a model that generalizes well. See how all the pieces are beginning to fit together!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9f4gKX77FmYf"
      },
      "source": [
        "---\r\n",
        "# Section 3: Understanding BackProp in CNNs\r\n",
        "<br>\r\n",
        "\r\n",
        "*Estimated Completion Time: 95 mins from start of the session*"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lou3C9z5Ckyr",
        "cellView": "form"
      },
      "source": [
        "#@title Video:Maxpool and Backprop on Conv Layers\r\n",
        "try: t4;\r\n",
        "except NameError: t4=time.time()\r\n",
        "\r\n",
        "video = YouTubeVideo(id=\"7aJywqGiR6U\", width=854, height=480, fs=1)\r\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\r\n",
        "\r\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lvt3cHM7G6br",
        "cellView": "form"
      },
      "source": [
        "#@markdown ## Exercise 3\n",
        "#@markdown How do you think gradients are affected when using maxpool, stride?\n",
        "back_prop = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mYdpbWWiFsjK"
      },
      "source": [
        "---\n",
        "# Section 4: What do CNNs Learn?\n",
        "\n",
        "At this point, you've trained a CNN and understand backpropagation. Now, we'd like to understand a bit more about what exactly do the layers of a CNN learn.\n",
        "\n",
        "*Estimated Completion Time: 110 mins from start of the session*"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KwgmzbPaD8v5",
        "cellView": "form"
      },
      "source": [
        "#@title Video: Visualizing Training curves\r\n",
        "try: t5;\r\n",
        "except NameError: t5=time.time()\r\n",
        "\r\n",
        "video = YouTubeVideo(id=\"y_gCwSyTKek\", width=854, height=480, fs=1)\r\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\r\n",
        "\r\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YD698xNkLbvK",
        "cellView": "form"
      },
      "source": [
        "##@title Download and set up Imagenette (A small version of Imagenet)\n",
        "#@title Download and set up Image for Visualisation\n",
        "\n",
        "# import os\n",
        "# os.system('rm -r imagenette*')\n",
        "# print(\"Downloading Imagenette ...\")\n",
        "# !wget https://s3.amazonaws.com/fast-ai-imageclas/imagenette2.tgz\n",
        "# print(\"Download Complete ...\")\n",
        "# print(\"Unpacking ...\")\n",
        "# os.system('tar -xf imagenette2.tgz')\n",
        "# print(\"Removing Redundancies ...\")\n",
        "# os.system('rm -r imagenette2.tgz')\n",
        "# print(\"All Done !\")\n",
        "\n",
        "!curl https://raw.githubusercontent.com/CIS-522/course-content/main/tutorials/W06_ConvNets/static/chainsaw.JPEG -o image_test.png\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eMBlzMUZLeCb",
        "cellView": "form"
      },
      "source": [
        "#@title Set Up Textual ImageNet labels\n",
        "dict_map={0: 'tench, Tinca tinca',\n",
        " 1: 'goldfish, Carassius auratus',\n",
        " 2: 'great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias',\n",
        " 3: 'tiger shark, Galeocerdo cuvieri',\n",
        " 4: 'hammerhead, hammerhead shark',\n",
        " 5: 'electric ray, crampfish, numbfish, torpedo',\n",
        " 6: 'stingray',\n",
        " 7: 'cock',\n",
        " 8: 'hen',\n",
        " 9: 'ostrich, Struthio camelus',\n",
        " 10: 'brambling, Fringilla montifringilla',\n",
        " 11: 'goldfinch, Carduelis carduelis',\n",
        " 12: 'house finch, linnet, Carpodacus mexicanus',\n",
        " 13: 'junco, snowbird',\n",
        " 14: 'indigo bunting, indigo finch, indigo bird, Passerina cyanea',\n",
        " 15: 'robin, American robin, Turdus migratorius',\n",
        " 16: 'bulbul',\n",
        " 17: 'jay',\n",
        " 18: 'magpie',\n",
        " 19: 'chickadee',\n",
        " 20: 'water ouzel, dipper',\n",
        " 21: 'kite',\n",
        " 22: 'bald eagle, American eagle, Haliaeetus leucocephalus',\n",
        " 23: 'vulture',\n",
        " 24: 'great grey owl, great gray owl, Strix nebulosa',\n",
        " 25: 'European fire salamander, Salamandra salamandra',\n",
        " 26: 'common newt, Triturus vulgaris',\n",
        " 27: 'eft',\n",
        " 28: 'spotted salamander, Ambystoma maculatum',\n",
        " 29: 'axolotl, mud puppy, Ambystoma mexicanum',\n",
        " 30: 'bullfrog, Rana catesbeiana',\n",
        " 31: 'tree frog, tree-frog',\n",
        " 32: 'tailed frog, bell toad, ribbed toad, tailed toad, Ascaphus trui',\n",
        " 33: 'loggerhead, loggerhead turtle, Caretta caretta',\n",
        " 34: 'leatherback turtle, leatherback, leathery turtle, Dermochelys coriacea',\n",
        " 35: 'mud turtle',\n",
        " 36: 'terrapin',\n",
        " 37: 'box turtle, box tortoise',\n",
        " 38: 'banded gecko',\n",
        " 39: 'common iguana, iguana, Iguana iguana',\n",
        " 40: 'American chameleon, anole, Anolis carolinensis',\n",
        " 41: 'whiptail, whiptail lizard',\n",
        " 42: 'agama',\n",
        " 43: 'frilled lizard, Chlamydosaurus kingi',\n",
        " 44: 'alligator lizard',\n",
        " 45: 'Gila monster, Heloderma suspectum',\n",
        " 46: 'green lizard, Lacerta viridis',\n",
        " 47: 'African chameleon, Chamaeleo chamaeleon',\n",
        " 48: 'Komodo dragon, Komodo lizard, dragon lizard, giant lizard, Varanus komodoensis',\n",
        " 49: 'African crocodile, Nile crocodile, Crocodylus niloticus',\n",
        " 50: 'American alligator, Alligator mississipiensis',\n",
        " 51: 'triceratops',\n",
        " 52: 'thunder snake, worm snake, Carphophis amoenus',\n",
        " 53: 'ringneck snake, ring-necked snake, ring snake',\n",
        " 54: 'hognose snake, puff adder, sand viper',\n",
        " 55: 'green snake, grass snake',\n",
        " 56: 'king snake, kingsnake',\n",
        " 57: 'garter snake, grass snake',\n",
        " 58: 'water snake',\n",
        " 59: 'vine snake',\n",
        " 60: 'night snake, Hypsiglena torquata',\n",
        " 61: 'boa constrictor, Constrictor constrictor',\n",
        " 62: 'rock python, rock snake, Python sebae',\n",
        " 63: 'Indian cobra, Naja naja',\n",
        " 64: 'green mamba',\n",
        " 65: 'sea snake',\n",
        " 66: 'horned viper, cerastes, sand viper, horned asp, Cerastes cornutus',\n",
        " 67: 'diamondback, diamondback rattlesnake, Crotalus adamanteus',\n",
        " 68: 'sidewinder, horned rattlesnake, Crotalus cerastes',\n",
        " 69: 'trilobite',\n",
        " 70: 'harvestman, daddy longlegs, Phalangium opilio',\n",
        " 71: 'scorpion',\n",
        " 72: 'black and gold garden spider, Argiope aurantia',\n",
        " 73: 'barn spider, Araneus cavaticus',\n",
        " 74: 'garden spider, Aranea diademata',\n",
        " 75: 'black widow, Latrodectus mactans',\n",
        " 76: 'tarantula',\n",
        " 77: 'wolf spider, hunting spider',\n",
        " 78: 'tick',\n",
        " 79: 'centipede',\n",
        " 80: 'black grouse',\n",
        " 81: 'ptarmigan',\n",
        " 82: 'ruffed grouse, partridge, Bonasa umbellus',\n",
        " 83: 'prairie chicken, prairie grouse, prairie fowl',\n",
        " 84: 'peacock',\n",
        " 85: 'quail',\n",
        " 86: 'partridge',\n",
        " 87: 'African grey, African gray, Psittacus erithacus',\n",
        " 88: 'macaw',\n",
        " 89: 'sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita',\n",
        " 90: 'lorikeet',\n",
        " 91: 'coucal',\n",
        " 92: 'bee eater',\n",
        " 93: 'hornbill',\n",
        " 94: 'hummingbird',\n",
        " 95: 'jacamar',\n",
        " 96: 'toucan',\n",
        " 97: 'drake',\n",
        " 98: 'red-breasted merganser, Mergus serrator',\n",
        " 99: 'goose',\n",
        " 100: 'black swan, Cygnus atratus',\n",
        " 101: 'tusker',\n",
        " 102: 'echidna, spiny anteater, anteater',\n",
        " 103: 'platypus, duckbill, duckbilled platypus, duck-billed platypus, Ornithorhynchus anatinus',\n",
        " 104: 'wallaby, brush kangaroo',\n",
        " 105: 'koala, koala bear, kangaroo bear, native bear, Phascolarctos cinereus',\n",
        " 106: 'wombat',\n",
        " 107: 'jellyfish',\n",
        " 108: 'sea anemone, anemone',\n",
        " 109: 'brain coral',\n",
        " 110: 'flatworm, platyhelminth',\n",
        " 111: 'nematode, nematode worm, roundworm',\n",
        " 112: 'conch',\n",
        " 113: 'snail',\n",
        " 114: 'slug',\n",
        " 115: 'sea slug, nudibranch',\n",
        " 116: 'chiton, coat-of-mail shell, sea cradle, polyplacophore',\n",
        " 117: 'chambered nautilus, pearly nautilus, nautilus',\n",
        " 118: 'Dungeness crab, Cancer magister',\n",
        " 119: 'rock crab, Cancer irroratus',\n",
        " 120: 'fiddler crab',\n",
        " 121: 'king crab, Alaska crab, Alaskan king crab, Alaska king crab, Paralithodes camtschatica',\n",
        " 122: 'American lobster, Northern lobster, Maine lobster, Homarus americanus',\n",
        " 123: 'spiny lobster, langouste, rock lobster, crawfish, crayfish, sea crawfish',\n",
        " 124: 'crayfish, crawfish, crawdad, crawdaddy',\n",
        " 125: 'hermit crab',\n",
        " 126: 'isopod',\n",
        " 127: 'white stork, Ciconia ciconia',\n",
        " 128: 'black stork, Ciconia nigra',\n",
        " 129: 'spoonbill',\n",
        " 130: 'flamingo',\n",
        " 131: 'little blue heron, Egretta caerulea',\n",
        " 132: 'American egret, great white heron, Egretta albus',\n",
        " 133: 'bittern',\n",
        " 134: 'crane',\n",
        " 135: 'limpkin, Aramus pictus',\n",
        " 136: 'European gallinule, Porphyrio porphyrio',\n",
        " 137: 'American coot, marsh hen, mud hen, water hen, Fulica americana',\n",
        " 138: 'bustard',\n",
        " 139: 'ruddy turnstone, Arenaria interpres',\n",
        " 140: 'red-backed sandpiper, dunlin, Erolia alpina',\n",
        " 141: 'redshank, Tringa totanus',\n",
        " 142: 'dowitcher',\n",
        " 143: 'oystercatcher, oyster catcher',\n",
        " 144: 'pelican',\n",
        " 145: 'king penguin, Aptenodytes patagonica',\n",
        " 146: 'albatross, mollymawk',\n",
        " 147: 'grey whale, gray whale, devilfish, Eschrichtius gibbosus, Eschrichtius robustus',\n",
        " 148: 'killer whale, killer, orca, grampus, sea wolf, Orcinus orca',\n",
        " 149: 'dugong, Dugong dugon',\n",
        " 150: 'sea lion',\n",
        " 151: 'Chihuahua',\n",
        " 152: 'Japanese spaniel',\n",
        " 153: 'Maltese dog, Maltese terrier, Maltese',\n",
        " 154: 'Pekinese, Pekingese, Peke',\n",
        " 155: 'Shih-Tzu',\n",
        " 156: 'Blenheim spaniel',\n",
        " 157: 'papillon',\n",
        " 158: 'toy terrier',\n",
        " 159: 'Rhodesian ridgeback',\n",
        " 160: 'Afghan hound, Afghan',\n",
        " 161: 'basset, basset hound',\n",
        " 162: 'beagle',\n",
        " 163: 'bloodhound, sleuthhound',\n",
        " 164: 'bluetick',\n",
        " 165: 'black-and-tan coonhound',\n",
        " 166: 'Walker hound, Walker foxhound',\n",
        " 167: 'English foxhound',\n",
        " 168: 'redbone',\n",
        " 169: 'borzoi, Russian wolfhound',\n",
        " 170: 'Irish wolfhound',\n",
        " 171: 'Italian greyhound',\n",
        " 172: 'whippet',\n",
        " 173: 'Ibizan hound, Ibizan Podenco',\n",
        " 174: 'Norwegian elkhound, elkhound',\n",
        " 175: 'otterhound, otter hound',\n",
        " 176: 'Saluki, gazelle hound',\n",
        " 177: 'Scottish deerhound, deerhound',\n",
        " 178: 'Weimaraner',\n",
        " 179: 'Staffordshire bullterrier, Staffordshire bull terrier',\n",
        " 180: 'American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier',\n",
        " 181: 'Bedlington terrier',\n",
        " 182: 'Border terrier',\n",
        " 183: 'Kerry blue terrier',\n",
        " 184: 'Irish terrier',\n",
        " 185: 'Norfolk terrier',\n",
        " 186: 'Norwich terrier',\n",
        " 187: 'Yorkshire terrier',\n",
        " 188: 'wire-haired fox terrier',\n",
        " 189: 'Lakeland terrier',\n",
        " 190: 'Sealyham terrier, Sealyham',\n",
        " 191: 'Airedale, Airedale terrier',\n",
        " 192: 'cairn, cairn terrier',\n",
        " 193: 'Australian terrier',\n",
        " 194: 'Dandie Dinmont, Dandie Dinmont terrier',\n",
        " 195: 'Boston bull, Boston terrier',\n",
        " 196: 'miniature schnauzer',\n",
        " 197: 'giant schnauzer',\n",
        " 198: 'standard schnauzer',\n",
        " 199: 'Scotch terrier, Scottish terrier, Scottie',\n",
        " 200: 'Tibetan terrier, chrysanthemum dog',\n",
        " 201: 'silky terrier, Sydney silky',\n",
        " 202: 'soft-coated wheaten terrier',\n",
        " 203: 'West Highland white terrier',\n",
        " 204: 'Lhasa, Lhasa apso',\n",
        " 205: 'flat-coated retriever',\n",
        " 206: 'curly-coated retriever',\n",
        " 207: 'golden retriever',\n",
        " 208: 'Labrador retriever',\n",
        " 209: 'Chesapeake Bay retriever',\n",
        " 210: 'German short-haired pointer',\n",
        " 211: 'vizsla, Hungarian pointer',\n",
        " 212: 'English setter',\n",
        " 213: 'Irish setter, red setter',\n",
        " 214: 'Gordon setter',\n",
        " 215: 'Brittany spaniel',\n",
        " 216: 'clumber, clumber spaniel',\n",
        " 217: 'English springer, English springer spaniel',\n",
        " 218: 'Welsh springer spaniel',\n",
        " 219: 'cocker spaniel, English cocker spaniel, cocker',\n",
        " 220: 'Sussex spaniel',\n",
        " 221: 'Irish water spaniel',\n",
        " 222: 'kuvasz',\n",
        " 223: 'schipperke',\n",
        " 224: 'groenendael',\n",
        " 225: 'malinois',\n",
        " 226: 'briard',\n",
        " 227: 'kelpie',\n",
        " 228: 'komondor',\n",
        " 229: 'Old English sheepdog, bobtail',\n",
        " 230: 'Shetland sheepdog, Shetland sheep dog, Shetland',\n",
        " 231: 'collie',\n",
        " 232: 'Border collie',\n",
        " 233: 'Bouvier des Flandres, Bouviers des Flandres',\n",
        " 234: 'Rottweiler',\n",
        " 235: 'German shepherd, German shepherd dog, German police dog, alsatian',\n",
        " 236: 'Doberman, Doberman pinscher',\n",
        " 237: 'miniature pinscher',\n",
        " 238: 'Greater Swiss Mountain dog',\n",
        " 239: 'Bernese mountain dog',\n",
        " 240: 'Appenzeller',\n",
        " 241: 'EntleBucher',\n",
        " 242: 'boxer',\n",
        " 243: 'bull mastiff',\n",
        " 244: 'Tibetan mastiff',\n",
        " 245: 'French bulldog',\n",
        " 246: 'Great Dane',\n",
        " 247: 'Saint Bernard, St Bernard',\n",
        " 248: 'Eskimo dog, husky',\n",
        " 249: 'malamute, malemute, Alaskan malamute',\n",
        " 250: 'Siberian husky',\n",
        " 251: 'dalmatian, coach dog, carriage dog',\n",
        " 252: 'affenpinscher, monkey pinscher, monkey dog',\n",
        " 253: 'basenji',\n",
        " 254: 'pug, pug-dog',\n",
        " 255: 'Leonberg',\n",
        " 256: 'Newfoundland, Newfoundland dog',\n",
        " 257: 'Great Pyrenees',\n",
        " 258: 'Samoyed, Samoyede',\n",
        " 259: 'Pomeranian',\n",
        " 260: 'chow, chow chow',\n",
        " 261: 'keeshond',\n",
        " 262: 'Brabancon griffon',\n",
        " 263: 'Pembroke, Pembroke Welsh corgi',\n",
        " 264: 'Cardigan, Cardigan Welsh corgi',\n",
        " 265: 'toy poodle',\n",
        " 266: 'miniature poodle',\n",
        " 267: 'standard poodle',\n",
        " 268: 'Mexican hairless',\n",
        " 269: 'timber wolf, grey wolf, gray wolf, Canis lupus',\n",
        " 270: 'white wolf, Arctic wolf, Canis lupus tundrarum',\n",
        " 271: 'red wolf, maned wolf, Canis rufus, Canis niger',\n",
        " 272: 'coyote, prairie wolf, brush wolf, Canis latrans',\n",
        " 273: 'dingo, warrigal, warragal, Canis dingo',\n",
        " 274: 'dhole, Cuon alpinus',\n",
        " 275: 'African hunting dog, hyena dog, Cape hunting dog, Lycaon pictus',\n",
        " 276: 'hyena, hyaena',\n",
        " 277: 'red fox, Vulpes vulpes',\n",
        " 278: 'kit fox, Vulpes macrotis',\n",
        " 279: 'Arctic fox, white fox, Alopex lagopus',\n",
        " 280: 'grey fox, gray fox, Urocyon cinereoargenteus',\n",
        " 281: 'tabby, tabby cat',\n",
        " 282: 'tiger cat',\n",
        " 283: 'Persian cat',\n",
        " 284: 'Siamese cat, Siamese',\n",
        " 285: 'Egyptian cat',\n",
        " 286: 'cougar, puma, catamount, mountain lion, painter, panther, Felis concolor',\n",
        " 287: 'lynx, catamount',\n",
        " 288: 'leopard, Panthera pardus',\n",
        " 289: 'snow leopard, ounce, Panthera uncia',\n",
        " 290: 'jaguar, panther, Panthera onca, Felis onca',\n",
        " 291: 'lion, king of beasts, Panthera leo',\n",
        " 292: 'tiger, Panthera tigris',\n",
        " 293: 'cheetah, chetah, Acinonyx jubatus',\n",
        " 294: 'brown bear, bruin, Ursus arctos',\n",
        " 295: 'American black bear, black bear, Ursus americanus, Euarctos americanus',\n",
        " 296: 'ice bear, polar bear, Ursus Maritimus, Thalarctos maritimus',\n",
        " 297: 'sloth bear, Melursus ursinus, Ursus ursinus',\n",
        " 298: 'mongoose',\n",
        " 299: 'meerkat, mierkat',\n",
        " 300: 'tiger beetle',\n",
        " 301: 'ladybug, ladybeetle, lady beetle, ladybird, ladybird beetle',\n",
        " 302: 'ground beetle, carabid beetle',\n",
        " 303: 'long-horned beetle, longicorn, longicorn beetle',\n",
        " 304: 'leaf beetle, chrysomelid',\n",
        " 305: 'dung beetle',\n",
        " 306: 'rhinoceros beetle',\n",
        " 307: 'weevil',\n",
        " 308: 'fly',\n",
        " 309: 'bee',\n",
        " 310: 'ant, emmet, pismire',\n",
        " 311: 'grasshopper, hopper',\n",
        " 312: 'cricket',\n",
        " 313: 'walking stick, walkingstick, stick insect',\n",
        " 314: 'cockroach, roach',\n",
        " 315: 'mantis, mantid',\n",
        " 316: 'cicada, cicala',\n",
        " 317: 'leafhopper',\n",
        " 318: 'lacewing, lacewing fly',\n",
        " 319: \"dragonfly, darning needle, devil's darning needle, sewing needle, snake feeder, snake doctor, mosquito hawk, skeeter hawk\",\n",
        " 320: 'damselfly',\n",
        " 321: 'admiral',\n",
        " 322: 'ringlet, ringlet butterfly',\n",
        " 323: 'monarch, monarch butterfly, milkweed butterfly, Danaus plexippus',\n",
        " 324: 'cabbage butterfly',\n",
        " 325: 'sulphur butterfly, sulfur butterfly',\n",
        " 326: 'lycaenid, lycaenid butterfly',\n",
        " 327: 'starfish, sea star',\n",
        " 328: 'sea urchin',\n",
        " 329: 'sea cucumber, holothurian',\n",
        " 330: 'wood rabbit, cottontail, cottontail rabbit',\n",
        " 331: 'hare',\n",
        " 332: 'Angora, Angora rabbit',\n",
        " 333: 'hamster',\n",
        " 334: 'porcupine, hedgehog',\n",
        " 335: 'fox squirrel, eastern fox squirrel, Sciurus niger',\n",
        " 336: 'marmot',\n",
        " 337: 'beaver',\n",
        " 338: 'guinea pig, Cavia cobaya',\n",
        " 339: 'sorrel',\n",
        " 340: 'zebra',\n",
        " 341: 'hog, pig, grunter, squealer, Sus scrofa',\n",
        " 342: 'wild boar, boar, Sus scrofa',\n",
        " 343: 'warthog',\n",
        " 344: 'hippopotamus, hippo, river horse, Hippopotamus amphibius',\n",
        " 345: 'ox',\n",
        " 346: 'water buffalo, water ox, Asiatic buffalo, Bubalus bubalis',\n",
        " 347: 'bison',\n",
        " 348: 'ram, tup',\n",
        " 349: 'bighorn, bighorn sheep, cimarron, Rocky Mountain bighorn, Rocky Mountain sheep, Ovis canadensis',\n",
        " 350: 'ibex, Capra ibex',\n",
        " 351: 'hartebeest',\n",
        " 352: 'impala, Aepyceros melampus',\n",
        " 353: 'gazelle',\n",
        " 354: 'Arabian camel, dromedary, Camelus dromedarius',\n",
        " 355: 'llama',\n",
        " 356: 'weasel',\n",
        " 357: 'mink',\n",
        " 358: 'polecat, fitch, foulmart, foumart, Mustela putorius',\n",
        " 359: 'black-footed ferret, ferret, Mustela nigripes',\n",
        " 360: 'otter',\n",
        " 361: 'skunk, polecat, wood pussy',\n",
        " 362: 'badger',\n",
        " 363: 'armadillo',\n",
        " 364: 'three-toed sloth, ai, Bradypus tridactylus',\n",
        " 365: 'orangutan, orang, orangutang, Pongo pygmaeus',\n",
        " 366: 'gorilla, Gorilla gorilla',\n",
        " 367: 'chimpanzee, chimp, Pan troglodytes',\n",
        " 368: 'gibbon, Hylobates lar',\n",
        " 369: 'siamang, Hylobates syndactylus, Symphalangus syndactylus',\n",
        " 370: 'guenon, guenon monkey',\n",
        " 371: 'patas, hussar monkey, Erythrocebus patas',\n",
        " 372: 'baboon',\n",
        " 373: 'macaque',\n",
        " 374: 'langur',\n",
        " 375: 'colobus, colobus monkey',\n",
        " 376: 'proboscis monkey, Nasalis larvatus',\n",
        " 377: 'marmoset',\n",
        " 378: 'capuchin, ringtail, Cebus capucinus',\n",
        " 379: 'howler monkey, howler',\n",
        " 380: 'titi, titi monkey',\n",
        " 381: 'spider monkey, Ateles geoffroyi',\n",
        " 382: 'squirrel monkey, Saimiri sciureus',\n",
        " 383: 'Madagascar cat, ring-tailed lemur, Lemur catta',\n",
        " 384: 'indri, indris, Indri indri, Indri brevicaudatus',\n",
        " 385: 'Indian elephant, Elephas maximus',\n",
        " 386: 'African elephant, Loxodonta africana',\n",
        " 387: 'lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens',\n",
        " 388: 'giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca',\n",
        " 389: 'barracouta, snoek',\n",
        " 390: 'eel',\n",
        " 391: 'coho, cohoe, coho salmon, blue jack, silver salmon, Oncorhynchus kisutch',\n",
        " 392: 'rock beauty, Holocanthus tricolor',\n",
        " 393: 'anemone fish',\n",
        " 394: 'sturgeon',\n",
        " 395: 'gar, garfish, garpike, billfish, Lepisosteus osseus',\n",
        " 396: 'lionfish',\n",
        " 397: 'puffer, pufferfish, blowfish, globefish',\n",
        " 398: 'abacus',\n",
        " 399: 'abaya',\n",
        " 400: \"academic gown, academic robe, judge's robe\",\n",
        " 401: 'accordion, piano accordion, squeeze box',\n",
        " 402: 'acoustic guitar',\n",
        " 403: 'aircraft carrier, carrier, flattop, attack aircraft carrier',\n",
        " 404: 'airliner',\n",
        " 405: 'airship, dirigible',\n",
        " 406: 'altar',\n",
        " 407: 'ambulance',\n",
        " 408: 'amphibian, amphibious vehicle',\n",
        " 409: 'analog clock',\n",
        " 410: 'apiary, bee house',\n",
        " 411: 'apron',\n",
        " 412: 'ashcan, trash can, garbage can, wastebin, ash bin, ash-bin, ashbin, dustbin, trash barrel, trash bin',\n",
        " 413: 'assault rifle, assault gun',\n",
        " 414: 'backpack, back pack, knapsack, packsack, rucksack, haversack',\n",
        " 415: 'bakery, bakeshop, bakehouse',\n",
        " 416: 'balance beam, beam',\n",
        " 417: 'balloon',\n",
        " 418: 'ballpoint, ballpoint pen, ballpen, Biro',\n",
        " 419: 'Band Aid',\n",
        " 420: 'banjo',\n",
        " 421: 'bannister, banister, balustrade, balusters, handrail',\n",
        " 422: 'barbell',\n",
        " 423: 'barber chair',\n",
        " 424: 'barbershop',\n",
        " 425: 'barn',\n",
        " 426: 'barometer',\n",
        " 427: 'barrel, cask',\n",
        " 428: 'barrow, garden cart, lawn cart, wheelbarrow',\n",
        " 429: 'baseball',\n",
        " 430: 'basketball',\n",
        " 431: 'bassinet',\n",
        " 432: 'bassoon',\n",
        " 433: 'bathing cap, swimming cap',\n",
        " 434: 'bath towel',\n",
        " 435: 'bathtub, bathing tub, bath, tub',\n",
        " 436: 'beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon',\n",
        " 437: 'beacon, lighthouse, beacon light, pharos',\n",
        " 438: 'beaker',\n",
        " 439: 'bearskin, busby, shako',\n",
        " 440: 'beer bottle',\n",
        " 441: 'beer glass',\n",
        " 442: 'bell cote, bell cot',\n",
        " 443: 'bib',\n",
        " 444: 'bicycle-built-for-two, tandem bicycle, tandem',\n",
        " 445: 'bikini, two-piece',\n",
        " 446: 'binder, ring-binder',\n",
        " 447: 'binoculars, field glasses, opera glasses',\n",
        " 448: 'birdhouse',\n",
        " 449: 'boathouse',\n",
        " 450: 'bobsled, bobsleigh, bob',\n",
        " 451: 'bolo tie, bolo, bola tie, bola',\n",
        " 452: 'bonnet, poke bonnet',\n",
        " 453: 'bookcase',\n",
        " 454: 'bookshop, bookstore, bookstall',\n",
        " 455: 'bottlecap',\n",
        " 456: 'bow',\n",
        " 457: 'bow tie, bow-tie, bowtie',\n",
        " 458: 'brass, memorial tablet, plaque',\n",
        " 459: 'brassiere, bra, bandeau',\n",
        " 460: 'breakwater, groin, groyne, mole, bulwark, seawall, jetty',\n",
        " 461: 'breastplate, aegis, egis',\n",
        " 462: 'broom',\n",
        " 463: 'bucket, pail',\n",
        " 464: 'buckle',\n",
        " 465: 'bulletproof vest',\n",
        " 466: 'bullet train, bullet',\n",
        " 467: 'butcher shop, meat market',\n",
        " 468: 'cab, hack, taxi, taxicab',\n",
        " 469: 'caldron, cauldron',\n",
        " 470: 'candle, taper, wax light',\n",
        " 471: 'cannon',\n",
        " 472: 'canoe',\n",
        " 473: 'can opener, tin opener',\n",
        " 474: 'cardigan',\n",
        " 475: 'car mirror',\n",
        " 476: 'carousel, carrousel, merry-go-round, roundabout, whirligig',\n",
        " 477: \"carpenter's kit, tool kit\",\n",
        " 478: 'carton',\n",
        " 479: 'car wheel',\n",
        " 480: 'cash machine, cash dispenser, automated teller machine, automatic teller machine, automated teller, automatic teller, ATM',\n",
        " 481: 'cassette',\n",
        " 482: 'cassette player',\n",
        " 483: 'castle',\n",
        " 484: 'catamaran',\n",
        " 485: 'CD player',\n",
        " 486: 'cello, violoncello',\n",
        " 487: 'cellular telephone, cellular phone, cellphone, cell, mobile phone',\n",
        " 488: 'chain',\n",
        " 489: 'chainlink fence',\n",
        " 490: 'chain mail, ring mail, mail, chain armor, chain armour, ring armor, ring armour',\n",
        " 491: 'chain saw, chainsaw',\n",
        " 492: 'chest',\n",
        " 493: 'chiffonier, commode',\n",
        " 494: 'chime, bell, gong',\n",
        " 495: 'china cabinet, china closet',\n",
        " 496: 'Christmas stocking',\n",
        " 497: 'church, church building',\n",
        " 498: 'cinema, movie theater, movie theatre, movie house, picture palace',\n",
        " 499: 'cleaver, meat cleaver, chopper',\n",
        " 500: 'cliff dwelling',\n",
        " 501: 'cloak',\n",
        " 502: 'clog, geta, patten, sabot',\n",
        " 503: 'cocktail shaker',\n",
        " 504: 'coffee mug',\n",
        " 505: 'coffeepot',\n",
        " 506: 'coil, spiral, volute, whorl, helix',\n",
        " 507: 'combination lock',\n",
        " 508: 'computer keyboard, keypad',\n",
        " 509: 'confectionery, confectionary, candy store',\n",
        " 510: 'container ship, containership, container vessel',\n",
        " 511: 'convertible',\n",
        " 512: 'corkscrew, bottle screw',\n",
        " 513: 'cornet, horn, trumpet, trump',\n",
        " 514: 'cowboy boot',\n",
        " 515: 'cowboy hat, ten-gallon hat',\n",
        " 516: 'cradle',\n",
        " 517: 'crane',\n",
        " 518: 'crash helmet',\n",
        " 519: 'crate',\n",
        " 520: 'crib, cot',\n",
        " 521: 'Crock Pot',\n",
        " 522: 'croquet ball',\n",
        " 523: 'crutch',\n",
        " 524: 'cuirass',\n",
        " 525: 'dam, dike, dyke',\n",
        " 526: 'desk',\n",
        " 527: 'desktop computer',\n",
        " 528: 'dial telephone, dial phone',\n",
        " 529: 'diaper, nappy, napkin',\n",
        " 530: 'digital clock',\n",
        " 531: 'digital watch',\n",
        " 532: 'dining table, board',\n",
        " 533: 'dishrag, dishcloth',\n",
        " 534: 'dishwasher, dish washer, dishwashing machine',\n",
        " 535: 'disk brake, disc brake',\n",
        " 536: 'dock, dockage, docking facility',\n",
        " 537: 'dogsled, dog sled, dog sleigh',\n",
        " 538: 'dome',\n",
        " 539: 'doormat, welcome mat',\n",
        " 540: 'drilling platform, offshore rig',\n",
        " 541: 'drum, membranophone, tympan',\n",
        " 542: 'drumstick',\n",
        " 543: 'dumbbell',\n",
        " 544: 'Dutch oven',\n",
        " 545: 'electric fan, blower',\n",
        " 546: 'electric guitar',\n",
        " 547: 'electric locomotive',\n",
        " 548: 'entertainment center',\n",
        " 549: 'envelope',\n",
        " 550: 'espresso maker',\n",
        " 551: 'face powder',\n",
        " 552: 'feather boa, boa',\n",
        " 553: 'file, file cabinet, filing cabinet',\n",
        " 554: 'fireboat',\n",
        " 555: 'fire engine, fire truck',\n",
        " 556: 'fire screen, fireguard',\n",
        " 557: 'flagpole, flagstaff',\n",
        " 558: 'flute, transverse flute',\n",
        " 559: 'folding chair',\n",
        " 560: 'football helmet',\n",
        " 561: 'forklift',\n",
        " 562: 'fountain',\n",
        " 563: 'fountain pen',\n",
        " 564: 'four-poster',\n",
        " 565: 'freight car',\n",
        " 566: 'French horn, horn',\n",
        " 567: 'frying pan, frypan, skillet',\n",
        " 568: 'fur coat',\n",
        " 569: 'garbage truck, dustcart',\n",
        " 570: 'gasmask, respirator, gas helmet',\n",
        " 571: 'gas pump, gasoline pump, petrol pump, island dispenser',\n",
        " 572: 'goblet',\n",
        " 573: 'go-kart',\n",
        " 574: 'golf ball',\n",
        " 575: 'golfcart, golf cart',\n",
        " 576: 'gondola',\n",
        " 577: 'gong, tam-tam',\n",
        " 578: 'gown',\n",
        " 579: 'grand piano, grand',\n",
        " 580: 'greenhouse, nursery, glasshouse',\n",
        " 581: 'grille, radiator grille',\n",
        " 582: 'grocery store, grocery, food market, market',\n",
        " 583: 'guillotine',\n",
        " 584: 'hair slide',\n",
        " 585: 'hair spray',\n",
        " 586: 'half track',\n",
        " 587: 'hammer',\n",
        " 588: 'hamper',\n",
        " 589: 'hand blower, blow dryer, blow drier, hair dryer, hair drier',\n",
        " 590: 'hand-held computer, hand-held microcomputer',\n",
        " 591: 'handkerchief, hankie, hanky, hankey',\n",
        " 592: 'hard disc, hard disk, fixed disk',\n",
        " 593: 'harmonica, mouth organ, harp, mouth harp',\n",
        " 594: 'harp',\n",
        " 595: 'harvester, reaper',\n",
        " 596: 'hatchet',\n",
        " 597: 'holster',\n",
        " 598: 'home theater, home theatre',\n",
        " 599: 'honeycomb',\n",
        " 600: 'hook, claw',\n",
        " 601: 'hoopskirt, crinoline',\n",
        " 602: 'horizontal bar, high bar',\n",
        " 603: 'horse cart, horse-cart',\n",
        " 604: 'hourglass',\n",
        " 605: 'iPod',\n",
        " 606: 'iron, smoothing iron',\n",
        " 607: \"jack-o'-lantern\",\n",
        " 608: 'jean, blue jean, denim',\n",
        " 609: 'jeep, landrover',\n",
        " 610: 'jersey, T-shirt, tee shirt',\n",
        " 611: 'jigsaw puzzle',\n",
        " 612: 'jinrikisha, ricksha, rickshaw',\n",
        " 613: 'joystick',\n",
        " 614: 'kimono',\n",
        " 615: 'knee pad',\n",
        " 616: 'knot',\n",
        " 617: 'lab coat, laboratory coat',\n",
        " 618: 'ladle',\n",
        " 619: 'lampshade, lamp shade',\n",
        " 620: 'laptop, laptop computer',\n",
        " 621: 'lawn mower, mower',\n",
        " 622: 'lens cap, lens cover',\n",
        " 623: 'letter opener, paper knife, paperknife',\n",
        " 624: 'library',\n",
        " 625: 'lifeboat',\n",
        " 626: 'lighter, light, igniter, ignitor',\n",
        " 627: 'limousine, limo',\n",
        " 628: 'liner, ocean liner',\n",
        " 629: 'lipstick, lip rouge',\n",
        " 630: 'Loafer',\n",
        " 631: 'lotion',\n",
        " 632: 'loudspeaker, speaker, speaker unit, loudspeaker system, speaker system',\n",
        " 633: \"loupe, jeweler's loupe\",\n",
        " 634: 'lumbermill, sawmill',\n",
        " 635: 'magnetic compass',\n",
        " 636: 'mailbag, postbag',\n",
        " 637: 'mailbox, letter box',\n",
        " 638: 'maillot',\n",
        " 639: 'maillot, tank suit',\n",
        " 640: 'manhole cover',\n",
        " 641: 'maraca',\n",
        " 642: 'marimba, xylophone',\n",
        " 643: 'mask',\n",
        " 644: 'matchstick',\n",
        " 645: 'maypole',\n",
        " 646: 'maze, labyrinth',\n",
        " 647: 'measuring cup',\n",
        " 648: 'medicine chest, medicine cabinet',\n",
        " 649: 'megalith, megalithic structure',\n",
        " 650: 'microphone, mike',\n",
        " 651: 'microwave, microwave oven',\n",
        " 652: 'military uniform',\n",
        " 653: 'milk can',\n",
        " 654: 'minibus',\n",
        " 655: 'miniskirt, mini',\n",
        " 656: 'minivan',\n",
        " 657: 'missile',\n",
        " 658: 'mitten',\n",
        " 659: 'mixing bowl',\n",
        " 660: 'mobile home, manufactured home',\n",
        " 661: 'Model T',\n",
        " 662: 'modem',\n",
        " 663: 'monastery',\n",
        " 664: 'monitor',\n",
        " 665: 'moped',\n",
        " 666: 'mortar',\n",
        " 667: 'mortarboard',\n",
        " 668: 'mosque',\n",
        " 669: 'mosquito net',\n",
        " 670: 'motor scooter, scooter',\n",
        " 671: 'mountain bike, all-terrain bike, off-roader',\n",
        " 672: 'mountain tent',\n",
        " 673: 'mouse, computer mouse',\n",
        " 674: 'mousetrap',\n",
        " 675: 'moving van',\n",
        " 676: 'muzzle',\n",
        " 677: 'nail',\n",
        " 678: 'neck brace',\n",
        " 679: 'necklace',\n",
        " 680: 'nipple',\n",
        " 681: 'notebook, notebook computer',\n",
        " 682: 'obelisk',\n",
        " 683: 'oboe, hautboy, hautbois',\n",
        " 684: 'ocarina, sweet potato',\n",
        " 685: 'odometer, hodometer, mileometer, milometer',\n",
        " 686: 'oil filter',\n",
        " 687: 'organ, pipe organ',\n",
        " 688: 'oscilloscope, scope, cathode-ray oscilloscope, CRO',\n",
        " 689: 'overskirt',\n",
        " 690: 'oxcart',\n",
        " 691: 'oxygen mask',\n",
        " 692: 'packet',\n",
        " 693: 'paddle, boat paddle',\n",
        " 694: 'paddlewheel, paddle wheel',\n",
        " 695: 'padlock',\n",
        " 696: 'paintbrush',\n",
        " 697: \"pajama, pyjama, pj's, jammies\",\n",
        " 698: 'palace',\n",
        " 699: 'panpipe, pandean pipe, syrinx',\n",
        " 700: 'paper towel',\n",
        " 701: 'parachute, chute',\n",
        " 702: 'parallel bars, bars',\n",
        " 703: 'park bench',\n",
        " 704: 'parking meter',\n",
        " 705: 'passenger car, coach, carriage',\n",
        " 706: 'patio, terrace',\n",
        " 707: 'pay-phone, pay-station',\n",
        " 708: 'pedestal, plinth, footstall',\n",
        " 709: 'pencil box, pencil case',\n",
        " 710: 'pencil sharpener',\n",
        " 711: 'perfume, essence',\n",
        " 712: 'Petri dish',\n",
        " 713: 'photocopier',\n",
        " 714: 'pick, plectrum, plectron',\n",
        " 715: 'pickelhaube',\n",
        " 716: 'picket fence, paling',\n",
        " 717: 'pickup, pickup truck',\n",
        " 718: 'pier',\n",
        " 719: 'piggy bank, penny bank',\n",
        " 720: 'pill bottle',\n",
        " 721: 'pillow',\n",
        " 722: 'ping-pong ball',\n",
        " 723: 'pinwheel',\n",
        " 724: 'pirate, pirate ship',\n",
        " 725: 'pitcher, ewer',\n",
        " 726: \"plane, carpenter's plane, woodworking plane\",\n",
        " 727: 'planetarium',\n",
        " 728: 'plastic bag',\n",
        " 729: 'plate rack',\n",
        " 730: 'plow, plough',\n",
        " 731: \"plunger, plumber's helper\",\n",
        " 732: 'Polaroid camera, Polaroid Land camera',\n",
        " 733: 'pole',\n",
        " 734: 'police van, police wagon, paddy wagon, patrol wagon, wagon, black Maria',\n",
        " 735: 'poncho',\n",
        " 736: 'pool table, billiard table, snooker table',\n",
        " 737: 'pop bottle, soda bottle',\n",
        " 738: 'pot, flowerpot',\n",
        " 739: \"potter's wheel\",\n",
        " 740: 'power drill',\n",
        " 741: 'prayer rug, prayer mat',\n",
        " 742: 'printer',\n",
        " 743: 'prison, prison house',\n",
        " 744: 'projectile, missile',\n",
        " 745: 'projector',\n",
        " 746: 'puck, hockey puck',\n",
        " 747: 'punching bag, punch bag, punching ball, punchball',\n",
        " 748: 'purse',\n",
        " 749: 'quill, quill pen',\n",
        " 750: 'quilt, comforter, comfort, puff',\n",
        " 751: 'racer, race car, racing car',\n",
        " 752: 'racket, racquet',\n",
        " 753: 'radiator',\n",
        " 754: 'radio, wireless',\n",
        " 755: 'radio telescope, radio reflector',\n",
        " 756: 'rain barrel',\n",
        " 757: 'recreational vehicle, RV, R.V.',\n",
        " 758: 'reel',\n",
        " 759: 'reflex camera',\n",
        " 760: 'refrigerator, icebox',\n",
        " 761: 'remote control, remote',\n",
        " 762: 'restaurant, eating house, eating place, eatery',\n",
        " 763: 'revolver, six-gun, six-shooter',\n",
        " 764: 'rifle',\n",
        " 765: 'rocking chair, rocker',\n",
        " 766: 'rotisserie',\n",
        " 767: 'rubber eraser, rubber, pencil eraser',\n",
        " 768: 'rugby ball',\n",
        " 769: 'rule, ruler',\n",
        " 770: 'running shoe',\n",
        " 771: 'safe',\n",
        " 772: 'safety pin',\n",
        " 773: 'saltshaker, salt shaker',\n",
        " 774: 'sandal',\n",
        " 775: 'sarong',\n",
        " 776: 'sax, saxophone',\n",
        " 777: 'scabbard',\n",
        " 778: 'scale, weighing machine',\n",
        " 779: 'school bus',\n",
        " 780: 'schooner',\n",
        " 781: 'scoreboard',\n",
        " 782: 'screen, CRT screen',\n",
        " 783: 'screw',\n",
        " 784: 'screwdriver',\n",
        " 785: 'seat belt, seatbelt',\n",
        " 786: 'sewing machine',\n",
        " 787: 'shield, buckler',\n",
        " 788: 'shoe shop, shoe-shop, shoe store',\n",
        " 789: 'shoji',\n",
        " 790: 'shopping basket',\n",
        " 791: 'shopping cart',\n",
        " 792: 'shovel',\n",
        " 793: 'shower cap',\n",
        " 794: 'shower curtain',\n",
        " 795: 'ski',\n",
        " 796: 'ski mask',\n",
        " 797: 'sleeping bag',\n",
        " 798: 'slide rule, slipstick',\n",
        " 799: 'sliding door',\n",
        " 800: 'slot, one-armed bandit',\n",
        " 801: 'snorkel',\n",
        " 802: 'snowmobile',\n",
        " 803: 'snowplow, snowplough',\n",
        " 804: 'soap dispenser',\n",
        " 805: 'soccer ball',\n",
        " 806: 'sock',\n",
        " 807: 'solar dish, solar collector, solar furnace',\n",
        " 808: 'sombrero',\n",
        " 809: 'soup bowl',\n",
        " 810: 'space bar',\n",
        " 811: 'space heater',\n",
        " 812: 'space shuttle',\n",
        " 813: 'spatula',\n",
        " 814: 'speedboat',\n",
        " 815: \"spider web, spider's web\",\n",
        " 816: 'spindle',\n",
        " 817: 'sports car, sport car',\n",
        " 818: 'spotlight, spot',\n",
        " 819: 'stage',\n",
        " 820: 'steam locomotive',\n",
        " 821: 'steel arch bridge',\n",
        " 822: 'steel drum',\n",
        " 823: 'stethoscope',\n",
        " 824: 'stole',\n",
        " 825: 'stone wall',\n",
        " 826: 'stopwatch, stop watch',\n",
        " 827: 'stove',\n",
        " 828: 'strainer',\n",
        " 829: 'streetcar, tram, tramcar, trolley, trolley car',\n",
        " 830: 'stretcher',\n",
        " 831: 'studio couch, day bed',\n",
        " 832: 'stupa, tope',\n",
        " 833: 'submarine, pigboat, sub, U-boat',\n",
        " 834: 'suit, suit of clothes',\n",
        " 835: 'sundial',\n",
        " 836: 'sunglass',\n",
        " 837: 'sunglasses, dark glasses, shades',\n",
        " 838: 'sunscreen, sunblock, sun blocker',\n",
        " 839: 'suspension bridge',\n",
        " 840: 'swab, swob, mop',\n",
        " 841: 'sweatshirt',\n",
        " 842: 'swimming trunks, bathing trunks',\n",
        " 843: 'swing',\n",
        " 844: 'switch, electric switch, electrical switch',\n",
        " 845: 'syringe',\n",
        " 846: 'table lamp',\n",
        " 847: 'tank, army tank, armored combat vehicle, armoured combat vehicle',\n",
        " 848: 'tape player',\n",
        " 849: 'teapot',\n",
        " 850: 'teddy, teddy bear',\n",
        " 851: 'television, television system',\n",
        " 852: 'tennis ball',\n",
        " 853: 'thatch, thatched roof',\n",
        " 854: 'theater curtain, theatre curtain',\n",
        " 855: 'thimble',\n",
        " 856: 'thresher, thrasher, threshing machine',\n",
        " 857: 'throne',\n",
        " 858: 'tile roof',\n",
        " 859: 'toaster',\n",
        " 860: 'tobacco shop, tobacconist shop, tobacconist',\n",
        " 861: 'toilet seat',\n",
        " 862: 'torch',\n",
        " 863: 'totem pole',\n",
        " 864: 'tow truck, tow car, wrecker',\n",
        " 865: 'toyshop',\n",
        " 866: 'tractor',\n",
        " 867: 'trailer truck, tractor trailer, trucking rig, rig, articulated lorry, semi',\n",
        " 868: 'tray',\n",
        " 869: 'trench coat',\n",
        " 870: 'tricycle, trike, velocipede',\n",
        " 871: 'trimaran',\n",
        " 872: 'tripod',\n",
        " 873: 'triumphal arch',\n",
        " 874: 'trolleybus, trolley coach, trackless trolley',\n",
        " 875: 'trombone',\n",
        " 876: 'tub, vat',\n",
        " 877: 'turnstile',\n",
        " 878: 'typewriter keyboard',\n",
        " 879: 'umbrella',\n",
        " 880: 'unicycle, monocycle',\n",
        " 881: 'upright, upright piano',\n",
        " 882: 'vacuum, vacuum cleaner',\n",
        " 883: 'vase',\n",
        " 884: 'vault',\n",
        " 885: 'velvet',\n",
        " 886: 'vending machine',\n",
        " 887: 'vestment',\n",
        " 888: 'viaduct',\n",
        " 889: 'violin, fiddle',\n",
        " 890: 'volleyball',\n",
        " 891: 'waffle iron',\n",
        " 892: 'wall clock',\n",
        " 893: 'wallet, billfold, notecase, pocketbook',\n",
        " 894: 'wardrobe, closet, press',\n",
        " 895: 'warplane, military plane',\n",
        " 896: 'washbasin, handbasin, washbowl, lavabo, wash-hand basin',\n",
        " 897: 'washer, automatic washer, washing machine',\n",
        " 898: 'water bottle',\n",
        " 899: 'water jug',\n",
        " 900: 'water tower',\n",
        " 901: 'whiskey jug',\n",
        " 902: 'whistle',\n",
        " 903: 'wig',\n",
        " 904: 'window screen',\n",
        " 905: 'window shade',\n",
        " 906: 'Windsor tie',\n",
        " 907: 'wine bottle',\n",
        " 908: 'wing',\n",
        " 909: 'wok',\n",
        " 910: 'wooden spoon',\n",
        " 911: 'wool, woolen, woollen',\n",
        " 912: 'worm fence, snake fence, snake-rail fence, Virginia fence',\n",
        " 913: 'wreck',\n",
        " 914: 'yawl',\n",
        " 915: 'yurt',\n",
        " 916: 'web site, website, internet site, site',\n",
        " 917: 'comic book',\n",
        " 918: 'crossword puzzle, crossword',\n",
        " 919: 'street sign',\n",
        " 920: 'traffic light, traffic signal, stoplight',\n",
        " 921: 'book jacket, dust cover, dust jacket, dust wrapper',\n",
        " 922: 'menu',\n",
        " 923: 'plate',\n",
        " 924: 'guacamole',\n",
        " 925: 'consomme',\n",
        " 926: 'hot pot, hotpot',\n",
        " 927: 'trifle',\n",
        " 928: 'ice cream, icecream',\n",
        " 929: 'ice lolly, lolly, lollipop, popsicle',\n",
        " 930: 'French loaf',\n",
        " 931: 'bagel, beigel',\n",
        " 932: 'pretzel',\n",
        " 933: 'cheeseburger',\n",
        " 934: 'hotdog, hot dog, red hot',\n",
        " 935: 'mashed potato',\n",
        " 936: 'head cabbage',\n",
        " 937: 'broccoli',\n",
        " 938: 'cauliflower',\n",
        " 939: 'zucchini, courgette',\n",
        " 940: 'spaghetti squash',\n",
        " 941: 'acorn squash',\n",
        " 942: 'butternut squash',\n",
        " 943: 'cucumber, cuke',\n",
        " 944: 'artichoke, globe artichoke',\n",
        " 945: 'bell pepper',\n",
        " 946: 'cardoon',\n",
        " 947: 'mushroom',\n",
        " 948: 'Granny Smith',\n",
        " 949: 'strawberry',\n",
        " 950: 'orange',\n",
        " 951: 'lemon',\n",
        " 952: 'fig',\n",
        " 953: 'pineapple, ananas',\n",
        " 954: 'banana',\n",
        " 955: 'jackfruit, jak, jack',\n",
        " 956: 'custard apple',\n",
        " 957: 'pomegranate',\n",
        " 958: 'hay',\n",
        " 959: 'carbonara',\n",
        " 960: 'chocolate sauce, chocolate syrup',\n",
        " 961: 'dough',\n",
        " 962: 'meat loaf, meatloaf',\n",
        " 963: 'pizza, pizza pie',\n",
        " 964: 'potpie',\n",
        " 965: 'burrito',\n",
        " 966: 'red wine',\n",
        " 967: 'espresso',\n",
        " 968: 'cup',\n",
        " 969: 'eggnog',\n",
        " 970: 'alp',\n",
        " 971: 'bubble',\n",
        " 972: 'cliff, drop, drop-off',\n",
        " 973: 'coral reef',\n",
        " 974: 'geyser',\n",
        " 975: 'lakeside, lakeshore',\n",
        " 976: 'promontory, headland, head, foreland',\n",
        " 977: 'sandbar, sand bar',\n",
        " 978: 'seashore, coast, seacoast, sea-coast',\n",
        " 979: 'valley, vale',\n",
        " 980: 'volcano',\n",
        " 981: 'ballplayer, baseball player',\n",
        " 982: 'groom, bridegroom',\n",
        " 983: 'scuba diver',\n",
        " 984: 'rapeseed',\n",
        " 985: 'daisy',\n",
        " 986: \"yellow lady's slipper, yellow lady-slipper, Cypripedium calceolus, Cypripedium parviflorum\",\n",
        " 987: 'corn',\n",
        " 988: 'acorn',\n",
        " 989: 'hip, rose hip, rosehip',\n",
        " 990: 'buckeye, horse chestnut, conker',\n",
        " 991: 'coral fungus',\n",
        " 992: 'agaric',\n",
        " 993: 'gyromitra',\n",
        " 994: 'stinkhorn, carrion fungus',\n",
        " 995: 'earthstar',\n",
        " 996: 'hen-of-the-woods, hen of the woods, Polyporus frondosus, Grifola frondosa',\n",
        " 997: 'bolete',\n",
        " 998: 'ear, spike, capitulum',\n",
        " 999: 'toilet tissue, toilet paper, bathroom tissue'}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "NjDiR8bKLjuj"
      },
      "source": [
        "#@title GradCam\n",
        "\n",
        "\n",
        "## Code from https://github.com/jacobgil/pytorch-grad-cam\n",
        "import torch\n",
        "import argparse\n",
        "import cv2\n",
        "import numpy as np\n",
        "import torch\n",
        "from torch.autograd import Function\n",
        "from torchvision import models, transforms\n",
        "\n",
        "class FeatureExtractor():\n",
        "    \"\"\" Class for extracting activations and\n",
        "    registering gradients from targetted intermediate layers \"\"\"\n",
        "\n",
        "    def __init__(self, model, target_layers):\n",
        "        self.model = model\n",
        "        self.target_layers = target_layers\n",
        "        self.gradients = []\n",
        "\n",
        "    def save_gradient(self, grad):\n",
        "        self.gradients.append(grad)\n",
        "\n",
        "    def __call__(self, x):\n",
        "        outputs = []\n",
        "        self.gradients = []\n",
        "        for name, module in self.model._modules.items():\n",
        "            x = module(x)\n",
        "            if name in self.target_layers:\n",
        "                x.register_hook(self.save_gradient)\n",
        "                outputs += [x]\n",
        "        return outputs, x\n",
        "\n",
        "class ModelOutputs():\n",
        "    \"\"\" Class for making a forward pass, and getting:\n",
        "    1. The network output.\n",
        "    2. Activations from intermeddiate targetted layers.\n",
        "    3. Gradients from intermeddiate targetted layers. \"\"\"\n",
        "\n",
        "    def __init__(self, model, feature_module, target_layers):\n",
        "        self.model = model\n",
        "        self.feature_module = feature_module\n",
        "        self.feature_extractor = FeatureExtractor(self.feature_module, target_layers)\n",
        "\n",
        "    def get_gradients(self):\n",
        "        return self.feature_extractor.gradients\n",
        "\n",
        "    def __call__(self, x):\n",
        "        target_activations = []\n",
        "        for name, module in self.model._modules.items():\n",
        "            if module == self.feature_module:\n",
        "                target_activations, x = self.feature_extractor(x)\n",
        "            elif \"avgpool\" in name.lower():\n",
        "                x = module(x)\n",
        "                x = x.view(x.size(0),-1)\n",
        "            else:\n",
        "                x = module(x)\n",
        "\n",
        "        return target_activations, x\n",
        "\n",
        "def preprocess_image(img):\n",
        "    normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],\n",
        "                                 std=[0.229, 0.224, 0.225])\n",
        "    preprocessing = transforms.Compose([\n",
        "        transforms.ToTensor(),\n",
        "        normalize,\n",
        "    ])\n",
        "    return preprocessing(img.copy()).unsqueeze(0)\n",
        "\n",
        "def show_cam_on_image(img, mask):\n",
        "    heatmap = cv2.applyColorMap(np.uint8(255 * mask), cv2.COLORMAP_JET)\n",
        "    heatmap = np.float32(heatmap) / 255\n",
        "    cam = heatmap + np.float32(img)\n",
        "    cam = cam / np.max(cam)\n",
        "    return np.uint8(255 * cam)\n",
        "\n",
        "class GradCam:\n",
        "    def __init__(self, model, feature_module, target_layer_names, use_cuda):\n",
        "        self.model = model\n",
        "        self.feature_module = feature_module\n",
        "        self.model.eval()\n",
        "        self.cuda = use_cuda\n",
        "        if self.cuda:\n",
        "            self.model = model.cuda()\n",
        "\n",
        "        self.extractor = ModelOutputs(self.model, self.feature_module, target_layer_names)\n",
        "\n",
        "    def forward(self, input_img):\n",
        "        return self.model(input_img)\n",
        "\n",
        "    def __call__(self, input_img, target_category=None):\n",
        "        if self.cuda:\n",
        "            input_img = input_img.cuda()\n",
        "\n",
        "        features, output = self.extractor(input_img)\n",
        "\n",
        "        if target_category == None:\n",
        "            target_category = np.argmax(output.cpu().data.numpy())\n",
        "\n",
        "        one_hot = np.zeros((1, output.size()[-1]), dtype=np.float32)\n",
        "        one_hot[0][target_category] = 1\n",
        "        one_hot = torch.from_numpy(one_hot).requires_grad_(True)\n",
        "        if self.cuda:\n",
        "            one_hot = one_hot.cuda()\n",
        "        \n",
        "        one_hot = torch.sum(one_hot * output)\n",
        "\n",
        "        self.feature_module.zero_grad()\n",
        "        self.model.zero_grad()\n",
        "        one_hot.backward(retain_graph=True)\n",
        "\n",
        "        grads_val = self.extractor.get_gradients()[-1].cpu().data.numpy()\n",
        "\n",
        "        target = features[-1]\n",
        "        target = target.cpu().data.numpy()[0, :]\n",
        "\n",
        "        weights = np.mean(grads_val, axis=(2, 3))[0, :]\n",
        "        cam = np.zeros(target.shape[1:], dtype=np.float32)\n",
        "\n",
        "        for i, w in enumerate(weights):\n",
        "            cam += w * target[i, :, :]\n",
        "\n",
        "        cam = np.maximum(cam, 0)\n",
        "        cam = cv2.resize(cam, input_img.shape[2:])\n",
        "        cam = cam - np.min(cam)\n",
        "        cam = cam / np.max(cam)\n",
        "        return cam\n",
        "\n",
        "\n",
        "class GuidedBackpropReLU(Function):\n",
        "    @staticmethod\n",
        "    def forward(self, input_img):\n",
        "        positive_mask = (input_img > 0).type_as(input_img)\n",
        "        output = torch.addcmul(torch.zeros(input_img.size()).type_as(input_img), input_img, positive_mask)\n",
        "        self.save_for_backward(input_img, output)\n",
        "        return output\n",
        "\n",
        "    @staticmethod\n",
        "    def backward(self, grad_output):\n",
        "        input_img, output = self.saved_tensors\n",
        "        grad_input = None\n",
        "\n",
        "        positive_mask_1 = (input_img > 0).type_as(grad_output)\n",
        "        positive_mask_2 = (grad_output > 0).type_as(grad_output)\n",
        "        grad_input = torch.addcmul(torch.zeros(input_img.size()).type_as(input_img),\n",
        "                                   torch.addcmul(torch.zeros(input_img.size()).type_as(input_img), grad_output,\n",
        "                                                 positive_mask_1), positive_mask_2)\n",
        "        return grad_input\n",
        "\n",
        "\n",
        "class GuidedBackpropReLUModel:\n",
        "    def __init__(self, model, use_cuda):\n",
        "        self.model = model\n",
        "        self.model.eval()\n",
        "        self.cuda = use_cuda\n",
        "        if self.cuda:\n",
        "            self.model = model.cuda()\n",
        "\n",
        "        def recursive_relu_apply(module_top):\n",
        "            for idx, module in module_top._modules.items():\n",
        "                recursive_relu_apply(module)\n",
        "                if module.__class__.__name__ == 'ReLU':\n",
        "                    module_top._modules[idx] = GuidedBackpropReLU.apply\n",
        "\n",
        "        # replace ReLU with GuidedBackpropReLU\n",
        "        recursive_relu_apply(self.model)\n",
        "\n",
        "    def forward(self, input_img):\n",
        "        return self.model(input_img)\n",
        "\n",
        "    def __call__(self, input_img, target_category=None):\n",
        "        if self.cuda:\n",
        "            input_img = input_img.cuda()\n",
        "\n",
        "        input_img = input_img.requires_grad_(True)\n",
        "\n",
        "        output = self.forward(input_img)\n",
        "\n",
        "        if target_category == None:\n",
        "            target_category = np.argmax(output.cpu().data.numpy())\n",
        "\n",
        "        one_hot = np.zeros((1, output.size()[-1]), dtype=np.float32)\n",
        "        one_hot[0][target_category] = 1\n",
        "        one_hot = torch.from_numpy(one_hot).requires_grad_(True)\n",
        "        if self.cuda:\n",
        "            one_hot = one_hot.cuda()\n",
        "\n",
        "        one_hot = torch.sum(one_hot * output)\n",
        "        one_hot.backward(retain_graph=True)\n",
        "\n",
        "        output = input_img.grad.cpu().data.numpy()\n",
        "        output = output[0, :, :, :]\n",
        "\n",
        "        return output\n",
        "\n",
        "def get_args():\n",
        "    parser = argparse.ArgumentParser()\n",
        "    parser.add_argument('--use-cuda', action='store_true', default=False,\n",
        "                        help='Use NVIDIA GPU acceleration')\n",
        "    parser.add_argument('--image-path', type=str, default='./examples/both.png',\n",
        "                        help='Input image path')\n",
        "    args = parser.parse_args()\n",
        "    args.use_cuda = args.use_cuda and torch.cuda.is_available()\n",
        "    if args.use_cuda:\n",
        "        print(\"Using GPU for acceleration\")\n",
        "    else:\n",
        "        print(\"Using CPU for computation\")\n",
        "\n",
        "    return args\n",
        "\n",
        "def deprocess_image(img):\n",
        "    \"\"\" see https://github.com/jacobgil/keras-grad-cam/blob/master/grad-cam.py#L65 \"\"\"\n",
        "    img = img - np.mean(img)\n",
        "    img = img / (np.std(img) + 1e-5)\n",
        "    img = img * 0.1\n",
        "    img = img + 0.5\n",
        "    img = np.clip(img, 0, 1)\n",
        "    return np.uint8(img*255)\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tqVbciwsLraF",
        "cellView": "form"
      },
      "source": [
        "#@title What does the network focus on ? \n",
        "from torchvision import models\n",
        "import matplotlib.pyplot as plt\n",
        "import torch\n",
        "dev=  torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
        "\n",
        "\n",
        "model = models.resnet50(pretrained=True)\n",
        "grad_cam = GradCam(model=model, feature_module=model.layer4, \\\n",
        "                    target_layer_names=[\"2\"], use_cuda=torch.cuda.is_available())\n",
        "model.eval()\n",
        "\n",
        "# k='./imagenette2'\n",
        "# k1=os.listdir(k)[1:]\n",
        "# k=k+'/'+k1[np.random.randint(len(k1))]\n",
        "# k1=os.listdir(k)\n",
        "# k=k+'/'+k1[np.random.randint(len(k1))]\n",
        "# k1=os.listdir(k)\n",
        "# k=k+'/'+k1[np.random.randint(len(k1))]\n",
        "\n",
        "k='image_test.png'\n",
        "img = cv2.imread(k, 1)\n",
        "\n",
        "fig=plt.figure()\n",
        "ax1 = fig.add_axes((0.1,0.4,0.8,0.5))\n",
        "ax1.set_title(\"Original Image\")\n",
        "ax1.imshow(img)\n",
        "fig.set_size_inches(7, 8, forward=True)\n",
        "plt.show()\n",
        "\n",
        "img_original=img.copy()\n",
        "#plt.imshow(cv2.cvtColor(img,cv2.COLOR_BGR2RGB))\n",
        "img = np.float32(img) / 255\n",
        "# Opencv loads as BGR:\n",
        "img = img[:, :, ::-1]\n",
        "input_img = preprocess_image(img)\n",
        "\n",
        "target_category = None\n",
        "grayscale_cam = grad_cam(input_img, target_category)\n",
        "\n",
        "grayscale_cam = cv2.resize(grayscale_cam, (img.shape[1], img.shape[0]))\n",
        "cam = show_cam_on_image(img, grayscale_cam)\n",
        "\n",
        "tens=model.to(dev)(input_img.to(dev))\n",
        "out=torch.softmax(tens,dim=1)\n",
        "_, indices = torch.sort(out, descending=True)\n",
        "\n",
        "\n",
        "fig=plt.figure()\n",
        "ax1 = fig.add_axes((0.1,0.4,0.8,0.5))\n",
        "ax1.set_title(\"Heatmap from the network\")\n",
        "ax1.imshow(cam)\n",
        "fig.set_size_inches(7, 8, forward=True)\n",
        "plt.show()\n",
        "\n",
        "from termcolor import colored\n",
        "#print(colored('hello', 'red'), colored('world', 'green'))\n",
        "print(colored(\"\\nClassification Results: \\n\",'green'))\n",
        "\n",
        "\n",
        "from prettytable import PrettyTable\n",
        "t = PrettyTable(['Class', 'Confidence'])\n",
        "out_l=[(dict_map[idx], 100*out[0][idx].item()) for idx in indices.cpu().numpy()[0][:5]]\n",
        "for i in out_l:\n",
        "    t.add_row([str(i[0]),i[1]])\n",
        "print(t)\n",
        "\n",
        "#tabulate(out_l,headers=['Class', 'Confidence'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SeSFCevmL5_s"
      },
      "source": [
        "We want you to play around with the widget below and look at what are the activations that the network is picking up on in the various layers. The features are being shown for the chainsaw image shown above.\n",
        "\n",
        "\n",
        "Discuss how you think a CNN learns."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bW8ytNuoLw-T",
        "cellView": "form"
      },
      "source": [
        "#@title What do the features look like ?\n",
        "Layer_Name= 'conv1' #@param [\"conv1\", \"layer1\",\"layer2\",\"layer3\",\"layer4\"]\n",
        "Num_Filters = 40 #@param {type:\"slider\", min:16, max:64, step:8}\n",
        "from skimage.transform import resize\n",
        "import matplotlib.pyplot as plt\n",
        "def temp_rem(model,key,input_img):\n",
        "    model.eval()\n",
        "    flag=0\n",
        "    alt_=[]    \n",
        "    for search in model._modules.keys():\n",
        "        if flag==1:\n",
        "            break\n",
        "        if key==search:\n",
        "            flag=1\n",
        "        \n",
        "        alt_.append(model._modules[search])\n",
        "\n",
        "    return torch.nn.Sequential(*alt_).to(dev)(input_img.to(dev)).squeeze().detach().cpu().numpy()\n",
        "\n",
        "def plot(data,Num_Filters,img_original):\n",
        "    import matplotlib.pyplot as plt\n",
        "    i=0\n",
        "    flag=0\n",
        "    if len(data)==3:\n",
        "        fig, ax = plt.subplots(nrows=1, ncols=4)\n",
        "\n",
        "        for col in ax:\n",
        "            if flag==0:\n",
        "                col.imshow(resize(img_original,(256,256)))    \n",
        "                flag=1\n",
        "            else:\n",
        "                col.imshow(resize(data[i],(256,256)))\n",
        "                i=i+1\n",
        "        fig.set_size_inches(15,15)\n",
        "        plt.tight_layout()\n",
        "        plt.subplots_adjust(wspace=0, hspace=0)\n",
        "        plt.show()\n",
        "    else:\n",
        "        ncols=8\n",
        "        nrows=Num_Filters//ncols\n",
        "        fig, ax = plt.subplots(nrows=nrows, ncols=ncols)\n",
        "        for row in ax:\n",
        "            for col in row:\n",
        "                if flag==0:\n",
        "                    col.imshow(resize(img_original,(256,256)))    \n",
        "                    col.set_xticklabels([])\n",
        "                    col.set_yticklabels([])\n",
        "                    #col.set_aspect('equal')\n",
        "                    flag=1\n",
        "                else:\n",
        "                    col.imshow(resize(data[i],(256,256)))\n",
        "                    col.set_xticklabels([])\n",
        "                    col.set_yticklabels([])\n",
        "                    #col.set_aspect('equal')\n",
        "                i=i+1\n",
        "        \n",
        "        \n",
        "        fig.set_size_inches(ncols*1.2,nrows*1.2)\n",
        "        plt.tight_layout()\n",
        "        plt.subplots_adjust(wspace=0, hspace=0)\n",
        "        plt.show()\n",
        "    \n",
        "\n",
        "\n",
        "\n",
        "#print(\"Named Blocks in ResNet50 {}\".format(model._modules.keys()))\n",
        "plot(temp_rem(model,Layer_Name,input_img),Num_Filters,img_original)\n",
        "#print(temp_rem(model,Layer_Name,input_img).shape)\n",
        "plt.show()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "owMg2eeTL0Hq"
      },
      "source": [
        "From the visualisations, we get a fair insight into how the features look. Careful observation brings us to the conclusion that while the initial layers and the initial convolutional block deals with the segregation of edge level data and the formation of feature maps with sharpened boundaries, as we move farther depthwise, the feature maps begin to catch on high level data, in terms of patterns in images and shapes of the objects.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lwk1Qc5svHSn"
      },
      "source": [
        "---\r\n",
        "# Section 5: Convolution Outside of Images\r\n",
        "\r\n",
        "\" A reminder that convnets are state of the art in various domains outside of image recognition.\"\r\n",
        "\r\n",
        "*Estimated Completion Time: 130 mins from start of the session*"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GAGwZu3BGAbQ",
        "cellView": "form"
      },
      "source": [
        "#@title Video: ConvNets Outside of Images\r\n",
        "try: t7;\r\n",
        "except NameError: t7=time.time()\r\n",
        "\r\n",
        "video = YouTubeVideo(id=\"91PkkHBnN4c\", width=854, height=480, fs=1)\r\n",
        "print(\"Video available at https://youtube.com/watch?v=\" + video.id)\r\n",
        "\r\n",
        "video"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3eqyA9UKpYGC"
      },
      "source": [
        "## Background\n",
        "\n",
        "The translational equivariance prior implemented by CNNs isn't only useful in images. There are a ton of fields where CNNs are used, but since [I'm](https://autobencoder.com/) a computational biologist, I'm going to talk about genomics.\n",
        "\n",
        "Specifically, this section demonstrates a model called [DeepSEA](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4768299/) that predicts transcription factor binding and epigenetic modification for a given DNA sequence.\n",
        "\n",
        "For people who don't speak geneticist, DNA is just the tip of the iceberg with respect to telling a cell what it should be doing. Proteins (known as transcription factors) and chemical groups (called epigenetics marks) regularly bind to DNA to cause various things to happen. If these modification happen in the wrong places, or at the wrong times, [diseases occur](https://www.nature.com/scitable/topicpage/epigenetic-influences-and-disease-895/).\n",
        "\n",
        "DeepSEA predicts which marks are likely given the DNA sequence in order to uncover the effects of genetic mutations. The authors use CNNs for this task because transcription factor binding and epigenetic effects are mostly dependent on the local sequence and not far-away sequences."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p1QH0mjoeXFP"
      },
      "source": [
        "model = kipoi.get_model('DeepSEA/predict')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "MhtigeS5osB4"
      },
      "source": [
        "# @title Download DNA Data\n",
        "# Download example dataloader kwargs\n",
        "dl_kwargs = model.default_dataloader.download_example('example')\n",
        "# Get the dataloader and instantiate it\n",
        "dl = model.default_dataloader(**dl_kwargs)\n",
        "# get a batch iterator\n",
        "it = dl.batch_iter(batch_size=10)\n",
        "# predict for a batch\n",
        "batch = next(it)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o0cJtJVyBrBJ"
      },
      "source": [
        "The next two cells show the input data (1000 base pair long DNA sequences)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nWw1sYuzrTrB"
      },
      "source": [
        "print(batch['inputs'].shape)\n",
        "example_sequence_one_hot = batch['inputs'][0,:,:,:].squeeze()\n",
        "print(example_sequence_one_hot.shape)\n",
        "example_sequence_indices = example_sequence_one_hot.argmax(axis=0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ubLUheqkrp4A"
      },
      "source": [
        "decoder = {0: 'A',\n",
        "           1: 'C',\n",
        "           2: 'G',\n",
        "           3: 'T'\n",
        "          }\n",
        "\n",
        "# Decode the DNA sequence\n",
        "decoded = []\n",
        "for index in example_sequence_indices:\n",
        "    decoded.append(decoder[index])\n",
        "    \n",
        "print(''.join(decoded))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GTyzG53YsGAY"
      },
      "source": [
        "# This file comes from the resources folder of the DeepSEA standalone package\n",
        "# that can be found here: http://deepsea.princeton.edu/media/code/deepsea.v0.94c.tar.gz\n",
        "!wget https://raw.githubusercontent.com/ben-heil/dl_workshop/main/notebooks/predictor.names"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-as489-TsGmI"
      },
      "source": [
        "with open('predictor.names') as feature_file:\n",
        "    feature_names = [name.strip() for name in feature_file]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5Q9FtwWOsJlN"
      },
      "source": [
        "pred = model.pipeline.predict(dl_kwargs, batch_size=10)\n",
        "pred.shape"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K16wT1uzsTaV"
      },
      "source": [
        "# Get the first prediction from the batch\n",
        "single_pred = torch.Tensor(pred[0,:])\n",
        "print(single_pred.shape)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ppss5NipsaUt"
      },
      "source": [
        "values, indices = torch.topk(single_pred, 5)\n",
        "\n",
        "for value, index in zip(values, indices):\n",
        "    feature = feature_names[index]\n",
        "    cell, mark, other = feature.split('|')\n",
        "    print('In cell line {}, feature {} has a probability of {}'.format(cell, mark, value))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HNfmO5cLtiXV"
      },
      "source": [
        "### Example Output Explained\n",
        "Three cell lines turn up in the top five results, but each one has the same epigenetic mark: Histone 3 K4 mono/dimethylation. This indicates that in those cell lines the gene downstream of this sequence is likely being actively transcribed into RNA."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0DDMxQ_1eePE"
      },
      "source": [
        "### Filter Visualization\n",
        "By incorporating domain knowledge, we can visualize and interpret filters used in genomics. The plot below (from [Ziga Avsec's tutorial](https://colab.research.google.com/github/Avsecz/DL-genomics-exercise/blob/master/Simulated.ipynb#scrollTo=aBM9vxtGZwmQ))shows filters learned by a CNN trained on DNA data. The size of each DNA base is proportional to the weight that the filter assigns to a given base at that position. Biologists viewing the digrams may be able to recognize [sequence motifs](https://en.wikipedia.org/wiki/Sequence_motif), specific genetic sequences targeted by various proteins."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "96Dn6qtZgjev"
      },
      "source": [
        "![logo.png]()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l7IH6ftpInad"
      },
      "source": [
        "In what other domains/problems might CNNs be useful? Why are CNNs effective in some cases but not others?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "so3sWAEUvKMp",
        "cellView": "form"
      },
      "source": [
        "other_cnn_domains = '' #@param {type:\"string\"}\n",
        "cnn_efficacy = '' #@param {type:\"string\"}\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uyHy54ZxPjPt"
      },
      "source": [
        "---\r\n",
        "# Submit your responses\r\n",
        "Please run the following cell and then press \"Submit\" so we can record your responses."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JAiufZ39Lpq8",
        "cellView": "form"
      },
      "source": [
        "import time\r\n",
        "import numpy as np\r\n",
        "import urllib.parse\r\n",
        "\r\n",
        "from IPython.display import IFrame\r\n",
        "#@markdown #Run Cell to Show Airtable Form\r\n",
        "#@markdown ##**Confirm your answers and then click \"Submit\"**\r\n",
        "\r\n",
        "def prefill_form(src, fields: dict):\r\n",
        "  '''\r\n",
        "  src: the original src url to embed the form\r\n",
        "  fields: a dictionary of field:value pairs,\r\n",
        "  e.g. {\"pennkey\": my_pennkey, \"location\": my_location}\r\n",
        "  '''\r\n",
        "  prefill_fields = {}\r\n",
        "  for key in fields:\r\n",
        "      new_key = 'prefill_' + key\r\n",
        "      prefill_fields[new_key] = fields[key]\r\n",
        "  prefills = urllib.parse.urlencode(prefill_fields)\r\n",
        "  src = src + prefills\r\n",
        "  return src\r\n",
        "\r\n",
        "\r\n",
        "#autofill time if it is not present\r\n",
        "try: t0;\r\n",
        "except NameError: t0 = time.time()\r\n",
        "try: t1;\r\n",
        "except NameError: t1 = time.time()\r\n",
        "try: t2;\r\n",
        "except NameError: t2 = time.time()\r\n",
        "try: t3;\r\n",
        "except NameError: t3 = time.time()\r\n",
        "try: t4;\r\n",
        "except NameError: t4 = time.time()\r\n",
        "try: t5;\r\n",
        "except NameError: t5 = time.time()\r\n",
        "try: t6;\r\n",
        "except NameError: t6 = time.time()\r\n",
        "try: t7;\r\n",
        "except NameError: t7 = time.time()\r\n",
        "try: t8;\r\n",
        "except NameError: t8 = time.time()\r\n",
        "\r\n",
        "#autofill fields if they are not present\r\n",
        "#a missing pennkey and pod will result in an Airtable warning\r\n",
        "#which is easily fixed user-side.\r\n",
        "try: my_pennkey;\r\n",
        "except NameError: my_pennkey = \"\"\r\n",
        "\r\n",
        "try: my_pod;\r\n",
        "except NameError: my_pod = \"Select\"\r\n",
        "\r\n",
        "try: output_dim;\r\n",
        "except NameError: output_dim = \"\"\r\n",
        "\r\n",
        "try: overfitting;\r\n",
        "except NameError: overfitting = \"\"\r\n",
        "\r\n",
        "try: regularize_1;\r\n",
        "except NameError: regularize_1 = \"\"\r\n",
        "\r\n",
        "try: regularize_2;\r\n",
        "except NameError: regularize_2 = \"\"\r\n",
        "\r\n",
        "try: augmentation;\r\n",
        "except NameError: augmentation = \"\"\r\n",
        "\r\n",
        "try: back_prop;\r\n",
        "except NameError: back_prop = \"\"\r\n",
        "\r\n",
        "try: other_cnn_domains;\r\n",
        "except NameError: other_cnn_domains = \"\"\r\n",
        "\r\n",
        "try: cnn_efficacy;\r\n",
        "except NameError: cnn_efficacy = \"\"\r\n",
        "\r\n",
        "times = np.array([t0,t1,t2,t3,t4,t5,t6,t7, t8])-t0\r\n",
        "\r\n",
        "fields = {\"my_pennkey\": my_pennkey,\r\n",
        "          \"my_pod\": my_pod,\r\n",
        "          \"output_dim\":output_dim,\r\n",
        "          \"overfitting\": overfitting,\r\n",
        "          \"regularize_1\":regularize_1,\r\n",
        "          \"regularize_2\": regularize_2,\r\n",
        "          \"augmentation\":augmentation,\r\n",
        "          \"back_prop\":back_prop,\r\n",
        "          \"other_cnn_domains\":other_cnn_domains,\r\n",
        "          \"cnn_efficacy\":cnn_efficacy,\r\n",
        "          \"cumulative_times\": times}\r\n",
        "\r\n",
        "src = \"https://airtable.com/embed/shruXwMiIqUw6vC8m?\"\r\n",
        "\r\n",
        "#now instead of the original source url, we do: src = prefill_form(src, fields)\r\n",
        "display(IFrame(src = prefill_form(src, fields), width = 800, height = 400))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0kglwCDy7x71"
      },
      "source": [
        "## Feedback\r\n",
        "How could this session have been better? How happy are you in your group? How do you feel right now?\r\n",
        "\r\n",
        "Feel free to use the embeded form below or use this link:\r\n",
        "<a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://airtable.com/shrNSJ5ECXhNhsYss\">https://airtable.com/shrNSJ5ECXhNhsYss</a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8_t2sxaejuMI"
      },
      "source": [
        "display(IFrame(src=\"https://airtable.com/embed/shrNSJ5ECXhNhsYss?backgroundColor=red\", width = 800, height = 400))"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}